-- Logs begin at Fri 2022-09-09 03:59:36 UTC, end at Fri 2022-09-09 04:59:22 UTC. -- Sep 09 04:24:24.214087 dv-cinder-856002-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 09 04:24:24.222144 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 09 04:24:24.224752 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Sep 9 04:24:24 2022] *** Sep 09 04:24:24.224872 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: nodename: dv-cinder-856002-3-iscsi Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: machine: x86_64 Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: clock source: unix Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: pcre jit disabled Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: detected number of CPU cores: 4 Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: current working directory: / Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: detected binary path: /usr/bin/uwsgi-core Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: your processes number limit is 47680 Sep 09 04:24:24.225010 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: your memory page size is 4096 bytes Sep 09 04:24:24.225551 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: detected max file descriptor number: 2048 Sep 09 04:24:24.225551 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: lock engine: pthread robust mutexes Sep 09 04:24:24.225551 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: thunder lock: enabled Sep 09 04:24:24.225551 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 09 04:24:24.225551 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 09 04:24:24.265218 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: Python main interpreter initialized at 0x560eef7429c0 Sep 09 04:24:24.265323 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: python threads support enabled Sep 09 04:24:24.265323 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: your server socket listen backlog is limited to 100 connections Sep 09 04:24:24.265323 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: your mercy for graceful operations on workers is 90 seconds Sep 09 04:24:24.265566 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: mapped 403077 bytes (393 KB) for 2 cores Sep 09 04:24:24.265730 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: *** Operational MODE: preforking *** Sep 09 04:24:24.265937 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: *** uWSGI is running in multiple interpreter mode *** Sep 09 04:24:24.265937 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: spawned uWSGI master process (pid: 81443) Sep 09 04:24:24.266130 dv-cinder-856002-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 09 04:24:24.266720 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: spawned uWSGI worker 1 (pid: 81451, cores: 1) Sep 09 04:24:24.267074 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: spawned uWSGI worker 2 (pid: 81452, cores: 1) Sep 09 04:24:24.267228 dv-cinder-856002-3-iscsi devstack@keystone.service[81443]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 09 04:24:28.525081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 09 04:24:28.525081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: from cryptography.utils import int_from_bytes Sep 09 04:24:28.525856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 09 04:24:28.525856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: from cryptography.utils import int_from_bytes Sep 09 04:24:28.526278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 09 04:24:28.526278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: from cryptography.utils import int_from_bytes Sep 09 04:24:28.526388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 09 04:24:28.526388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: from cryptography.utils import int_from_bytes Sep 09 04:24:28.575570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 09 04:24:28.575739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 09 04:24:28.575826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 09 04:24:28.575917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 09 04:24:28.575992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] command line args: [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 09 04:24:28.576063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] command line args: [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 09 04:24:28.576192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 09 04:24:28.576282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 09 04:24:28.576362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ================================================================================ {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 09 04:24:28.576470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ================================================================================ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 09 04:24:28.576611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] admin_token = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.576732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] admin_token = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.576858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] config_dir = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.576983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config_dir = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] config_source = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config_source = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] debug = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] debug = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.577991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.578986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_config_append = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_config_append = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_dir = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_dir = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_file = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.579928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.580960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.581975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.582896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.20/identity {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.20/identity {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] publish_errors = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] publish_errors = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.583900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.584934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] standard_threads = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] standard_threads = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_journal = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] use_journal = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.585945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] use_json = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_json = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] use_stderr = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_stderr = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] use_syslog = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_syslog = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 09 04:24:28.586732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.586808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.586991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.587980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.588958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.589921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.590899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591413 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.591978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.592977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.593922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.594083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.594653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.594828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.594913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.595894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.596986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.597981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598686 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.598917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.599929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.600945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.601976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.602867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.603879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.604847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.605963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.606918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.607985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.608948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.609876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.610456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.610610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.610692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.610846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.610924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.611907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.612879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.external = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.external = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.613837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.password = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.password = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] auth.token = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.614818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.token = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.615963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.616991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.617974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.618941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.619986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.620958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.621919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.622879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.623845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.624965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.625959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.626998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.627940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.628955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.629886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.630845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.631983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.632993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.633992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.634972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.635925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.636976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.637932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.638870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.639868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.640976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.641957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.642912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.643961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.644897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.645866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.646906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] role.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] role.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.647962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.648938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.649976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.650875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.651849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.652999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.653942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.654917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.655964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.656124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.656209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.656368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.656446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.656605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.657945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.658911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.659916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.660902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.connection = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.661855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.662967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.663926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.664943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.665100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.665179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 09 04:24:28.665251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 09 04:24:28.665322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 09 04:24:28.694570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.694721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.694803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.694877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.695049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.695135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.703401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.703988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.732406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.732522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.732643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.732721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.785881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.786293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.835099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.835334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.835864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.836106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.991885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.992981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.993063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.993195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:28.993412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.016598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.016836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.016922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.017949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.018031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.018160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.029204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.029405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.041971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.042965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.043104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.043189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.043323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.043411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.043546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 09 04:24:29.060328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.060542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 09 04:24:29.060863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.060948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.061077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 09 04:24:29.061177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 09 04:24:29.061352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.061436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.061607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 09 04:24:29.061690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.061873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.061957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.062138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.062219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.062397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.062656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.072724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.072821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.073005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.073090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.073276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.073373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.073559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.073643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.080574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.080691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.083481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.083608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.083772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.083903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.084902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.085879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.086002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.086158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.086241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.086433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.109385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.109582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.110956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.111133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.111251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.111425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.115982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.116095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.116270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.116386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.116553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.116674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.122951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.123042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.123246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.123329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.129343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.129441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.129644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.129730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.129936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.130885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.131090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.131173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.131359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.131440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.146728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.146824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.147032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.147118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.150806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.150903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.154007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.154092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.154307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.154390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.159030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.159125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.162130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.162214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.165377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.165477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.165682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.165773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.165966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.166951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.182673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.182786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.182987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.183114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.183260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.183389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.183523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.183655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.191134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.191361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.192675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.192917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.194190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.194449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.194533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.194731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.197165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.197453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.197538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.197744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.197826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.198028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.198113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.198310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.198422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.198697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.209628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.209915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.210000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.210198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.210336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.210478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.210574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.210779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.210863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.211156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.211252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.211447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.211531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.211744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.227342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.227683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.230235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.230601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.233151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.233555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.234577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.235062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.236114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.236400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.236550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.236795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.236972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.237198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.243153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.243599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.244616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.245058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.247520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.247910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.248038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.248197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.248384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.248533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.248667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.248945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.256753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.257044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.257326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.257477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.257694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.257888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.264202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.264489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.264771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.264899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 09 04:24:29.265044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.265179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.265315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.265448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.265714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.265939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.266745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.266996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.267943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.268284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.268284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.268598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.268733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.268892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.269032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.269163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.269163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.269163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 09 04:24:29.302263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x560eef7429c0 pid: 81451 (default app) Sep 09 04:24:29.302991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x560eef7429c0 pid: 81452 (default app) Sep 09 04:24:29.305532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b619746-219d-4d1c-91f2-960491458753 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:29.305821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b619746-219d-4d1c-91f2-960491458753 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:29.306095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b619746-219d-4d1c-91f2-960491458753 None None] PATH_INFO: `/v3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:29.310470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1/1] 10.222.0.20 () {50 vars in 791 bytes} [Fri Sep 9 04:24:29 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:36.033371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:36.033906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:36.033906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:36.038181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1/2] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:36 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:36.043528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:36.043990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:36.044129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:36.114882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: INFO dbcounter [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] Registered counter for database keystone Sep 09 04:24:36.133548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG oslo_db.sqlalchemy.engines [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81451) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 09 04:24:36.139854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writer thread running {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 09 04:24:36.316997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG passlib.handlers.bcrypt [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81451) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 09 04:24:36.317297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG passlib.handlers.bcrypt [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81451) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 09 04:24:36.374659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:36.376313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:36.503986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 2/3] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:36 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 461 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 09 04:24:36.948703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:37.018825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: INFO dbcounter [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] Registered counter for database keystone Sep 09 04:24:37.039856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG oslo_db.sqlalchemy.engines [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81452) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 09 04:24:37.046315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writer thread running {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 09 04:24:37.163433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cf9720-7a2b-4697-b5af-898e2fb926f4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:37.167670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:37.168297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:37.168579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:37.168851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:37.177464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:37.228798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING keystone.common.rbac_enforcer.enforcer [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 09 04:24:37.229091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033e6604-4cd5-40fd-b0f9-c360c64dec67 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:37.238868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 2/4] 10.222.0.20 () {58 vars in 1324 bytes} [Fri Sep 9 04:24:36 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:24:37.244696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:37.261891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b30cd-c9dc-4026-96c5-758d5da3a4ba None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:37.266777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:37.267925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:37.268207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:37.268486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] PATH_INFO: `/v3/projects/a501f44f388545198539783ea38ef108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:37.270071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] RBAC: Authorizing `identity:get_project(project_id=a501f44f388545198539783ea38ef108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:37.324081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING keystone.common.rbac_enforcer.enforcer [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 09 04:24:37.324421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e9c917-9355-4005-9cde-acccc9e8a4ba admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:37.329493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 3/5] 10.222.0.20 () {56 vars in 1207 bytes} [Fri Sep 9 04:24:37 2022] GET /identity/v3/projects/a501f44f388545198539783ea38ef108 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:39.431023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f46743-0770-4d9d-a9f6-5bf721609726 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:39.431580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f46743-0770-4d9d-a9f6-5bf721609726 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:39.431580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f46743-0770-4d9d-a9f6-5bf721609726 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:39.435929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 3/6] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:39.441348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:39.441748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:39.441900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:39.461982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:39.463512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:39.471888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 4/7] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:39 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 09 04:24:39.734127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f46743-0770-4d9d-a9f6-5bf721609726 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:39.746147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f46743-0770-4d9d-a9f6-5bf721609726 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:39.750789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9d21f7-c263-4e29-a6bb-c0438378551e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:39.751408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9d21f7-c263-4e29-a6bb-c0438378551e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:39.751688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9d21f7-c263-4e29-a6bb-c0438378551e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:39.751961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9d21f7-c263-4e29-a6bb-c0438378551e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:39.759498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9d21f7-c263-4e29-a6bb-c0438378551e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:39.761262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9d21f7-c263-4e29-a6bb-c0438378551e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:39.770501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 4/8] 10.222.0.20 () {58 vars in 1324 bytes} [Fri Sep 9 04:24:39 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:24:39.775642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:39.784636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9552f470-7195-4f31-9c88-0f6fbe07fe1f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:39.789053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abff8047-89a0-4bdd-b013-60d6e990588e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:39.789678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abff8047-89a0-4bdd-b013-60d6e990588e admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:39.789961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abff8047-89a0-4bdd-b013-60d6e990588e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:39.790245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abff8047-89a0-4bdd-b013-60d6e990588e admin admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:39.791598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abff8047-89a0-4bdd-b013-60d6e990588e admin admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:39.793421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abff8047-89a0-4bdd-b013-60d6e990588e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:39.794946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 5/9] 10.222.0.20 () {56 vars in 1198 bytes} [Fri Sep 9 04:24:39 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:42.034655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551cb563-bf99-4354-bc9c-ad710292d5aa None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.035230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551cb563-bf99-4354-bc9c-ad710292d5aa None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.035230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551cb563-bf99-4354-bc9c-ad710292d5aa None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.036577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 5/10] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:42.037617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9632be5b-b3fb-4635-98ff-c336fedaf567 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.038009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9632be5b-b3fb-4635-98ff-c336fedaf567 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.038170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9632be5b-b3fb-4635-98ff-c336fedaf567 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.041179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 6/11] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:24:42.042724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.043042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.043323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.046289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.046616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.046894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.065686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:42.066919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:42.097467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG passlib.handlers.bcrypt [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81452) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 09 04:24:42.097869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG passlib.handlers.bcrypt [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81452) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 09 04:24:42.146531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:42.147334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 7/12] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:24:42.148194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:42.249278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 6/13] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:42.417401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:42.442876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:42.521704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ded26cf-58d1-44ce-a805-0afc75fc1547 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:42.532500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:42.555760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:42.556367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.556645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.556918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.623868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:42.625821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:42.633046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3416193-985a-4799-b152-6cf4d732fefe None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:42.635358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:42.647976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:42.670778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:42.671904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.672192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.672466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.684288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:42.693279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:42.694351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:24:42.695208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 7/14] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:24:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:24:42.700961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:42.725769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 8/15] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:24:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:24:42.731294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:42.805253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1846b9fb-384f-4a40-bc16-6986bb6c9a1b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:42.827974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f689e63-9516-473c-baca-1479b4d8419b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:42.838105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:42.838716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.839032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.839312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.839983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:42.840288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:42.841726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:42.850612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:42.854212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 8/16] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:24:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:42.860367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:42.867793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:42.868431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.868714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.868989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.878886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:42.880390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:42.886811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:24:42.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:42.899435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:24:42.900232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:24:42.900232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:24:42.900232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:24:42.900232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 9/17] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:24:42.905352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:42.961342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e96caa4-4f7e-4341-9c52-2d239fbb25eb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:42.994865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:42.995548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:42.995829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:42.996102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:42.997574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:42.999116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:43.000789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 9/18] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:24:42 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:43.004447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe09224-6e6e-44a4-8588-6c5dad5a02f7 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:43.006231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:43.039729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:43.040337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:43.040622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:43.040904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:43.041554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:43.041861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:43.043297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:43.061487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 10/19] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:24:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:43.102608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:43.110320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a238f86d-4947-46c0-b812-d162598d93fd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:43.146291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45115c8-4926-4ed9-aa0e-cf1eaa7a0d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:43.146962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45115c8-4926-4ed9-aa0e-cf1eaa7a0d15 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:43.147246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45115c8-4926-4ed9-aa0e-cf1eaa7a0d15 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:43.147536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45115c8-4926-4ed9-aa0e-cf1eaa7a0d15 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:43.148678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45115c8-4926-4ed9-aa0e-cf1eaa7a0d15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:43.150164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45115c8-4926-4ed9-aa0e-cf1eaa7a0d15 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:43.151838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 10/20] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:43.158420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:43.195524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:43.265857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d628304b-78f1-4d79-9994-8a6a7f037011 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:43.299956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:43.300476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:43.300755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:43.301033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:43.301757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.domain.id=default, user.id=f2b9f22768104befbb8f56575b4c786b) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:43.302065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:43.303540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b682063a-2bc8-46c3-a68f-60c332bf1d9d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:43.311185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 11/21] 10.222.0.20 () {56 vars in 1322 bytes} [Fri Sep 9 04:24:43 2022] GET /identity/v3/role_assignments?user.id=f2b9f22768104befbb8f56575b4c786b&scope.domain.id=default&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 233 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:45.495804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1388c3ea-dc41-481b-9996-03650ab7d233 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:45.496309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1388c3ea-dc41-481b-9996-03650ab7d233 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:45.496410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1388c3ea-dc41-481b-9996-03650ab7d233 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:45.497876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 11/22] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:45.503292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8a2d6f-853b-4b91-b19b-be8745802eaa None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:45.503754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8a2d6f-853b-4b91-b19b-be8745802eaa None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:45.503840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8a2d6f-853b-4b91-b19b-be8745802eaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:45.532455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5c8a2d6f-853b-4b91-b19b-be8745802eaa None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:45.533601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5c8a2d6f-853b-4b91-b19b-be8745802eaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:45.617141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 12/23] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:45.707405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40c43a-4173-4fa7-87bb-febac0cacd55 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:45.708056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40c43a-4173-4fa7-87bb-febac0cacd55 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:45.708570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40c43a-4173-4fa7-87bb-febac0cacd55 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:45.710102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 12/24] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:24:45.715725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dcd894-0058-4ed0-8226-9fc9d99750f2 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:45.716016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dcd894-0058-4ed0-8226-9fc9d99750f2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:45.716297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dcd894-0058-4ed0-8226-9fc9d99750f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:45.744047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-37dcd894-0058-4ed0-8226-9fc9d99750f2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:45.745273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-37dcd894-0058-4ed0-8226-9fc9d99750f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:45.778866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:45.829928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 13/25] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:46.125236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40c43a-4173-4fa7-87bb-febac0cacd55 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:46.151807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f705a594-7298-4974-886f-156b8b3968aa None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.152424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f705a594-7298-4974-886f-156b8b3968aa None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.152888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f705a594-7298-4974-886f-156b8b3968aa None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.154690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 14/26] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:46.158421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f705a594-7298-4974-886f-156b8b3968aa None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:46.277596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f705a594-7298-4974-886f-156b8b3968aa None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:46.302357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40c43a-4173-4fa7-87bb-febac0cacd55 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:46.326583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:46.327607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.328521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.328941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.329786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:46.331398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:46.383116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:46.383726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.384015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.384295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.421087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:46.422648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d3f2bc59-a8aa-45c9-b7bd-4198e8bd4bb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:24:46.453587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:46.454712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:24:46.455739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:24:46.455739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:24:46.455739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:24:46.455739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 13/27] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:24:46.464169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff437a6-e64e-4641-b825-81acfc17f39c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.464467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff437a6-e64e-4641-b825-81acfc17f39c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.464746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff437a6-e64e-4641-b825-81acfc17f39c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.474436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 15/28] 10.222.0.20 () {60 vars in 1131 bytes} [Fri Sep 9 04:24:46 2022] POST /identity/v3/roles => generated 219 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:24:46.475699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:46.501752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0ff437a6-e64e-4641-b825-81acfc17f39c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:46.503687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0ff437a6-e64e-4641-b825-81acfc17f39c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:46.515482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:46.527595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:46.586562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 14/29] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:46.588257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d97bb-b0d5-405a-98d4-552ab01b43da None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.588697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d97bb-b0d5-405a-98d4-552ab01b43da None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.589116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d97bb-b0d5-405a-98d4-552ab01b43da None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.590631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 15/30] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:46.600237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.600556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.600784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.643190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:46.644350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:46.730753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bfe8a8-8340-4a22-988b-9f12cdbb303e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:46.776198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:46.777259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.777756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.778276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.779299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:46.779828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:46.781863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:46.799301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 16/31] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:24:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:46.807950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:46.810920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 16/32] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:46.898356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:46.927629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f57a0-5588-4a70-a092-1fb96a3eb130 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:46.950907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:46.971708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:46.972342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:46.972629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:46.973027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:46.973765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:46.975695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:47.018956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 17/33] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:24:46 2022] POST /identity/v3/projects => generated 301 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:24:47.080093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:47.109060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05d2c93-674a-4338-8fa5-a78a94535640 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:47.147117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:47.147832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.148120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.148460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.160457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:47.165460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:47.167384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:47.168219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:47.168953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:24:47.170965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 17/34] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:24:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:24:47.176938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:47.348050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8bb42b-7299-4c04-885d-45d115f6565c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:47.371326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ac617-4df1-4975-ba6a-a98d0e1700b4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:47.405157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:47.406232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.406514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.406789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.407504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:47.407806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:47.409200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:47.410616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:47.413677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:47.420762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 18/35] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:24:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:47.423659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d015bc71-0e1f-4c4c-8afc-51915261c367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:47.424382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d015bc71-0e1f-4c4c-8afc-51915261c367 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.424830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d015bc71-0e1f-4c4c-8afc-51915261c367 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.425227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d015bc71-0e1f-4c4c-8afc-51915261c367 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.425975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d015bc71-0e1f-4c4c-8afc-51915261c367 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:47.428046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:47.429115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d015bc71-0e1f-4c4c-8afc-51915261c367 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:47.450507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 18/36] 10.222.0.20 () {60 vars in 1131 bytes} [Fri Sep 9 04:24:47 2022] POST /identity/v3/roles => generated 225 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:24:47.481367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15cb5c0-775f-4607-a26c-1f9f1d85f2ad None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.481663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15cb5c0-775f-4607-a26c-1f9f1d85f2ad None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.481940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15cb5c0-775f-4607-a26c-1f9f1d85f2ad None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.484313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 19/37] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:47.490365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceeeb93-843d-41b8-ba9d-500902167a88 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.491856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceeeb93-843d-41b8-ba9d-500902167a88 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.492222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceeeb93-843d-41b8-ba9d-500902167a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.520016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-fceeeb93-843d-41b8-ba9d-500902167a88 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:47.522360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-fceeeb93-843d-41b8-ba9d-500902167a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:47.572138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5aaf4d-5c5f-42b7-bbbb-e7f98b7465aa None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:47.597027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 20/38] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:47.599164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e9708-19f7-4e1c-b53b-3488ab6b9af0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.599595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e9708-19f7-4e1c-b53b-3488ab6b9af0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.600004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e9708-19f7-4e1c-b53b-3488ab6b9af0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.601509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 21/39] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:47.614794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.615377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.615806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.618793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:47.619582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.619864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.620141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.621438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:47.623099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:47.626288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 19/40] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:24:47 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:47.632063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:47.636817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:47.639878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:47.721189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:47.727066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 22/41] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:24:47.765418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c91536-09ba-4fcc-ac4f-5badec9be60b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:47.799796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:47.800419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:47.800699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:47.800973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:47.802089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:47.803555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:47.805258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 20/42] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:47.811158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:47.870994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:47.967288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:47.983098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c23a705-14eb-4e20-8ddf-e3fd2d960faf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:48.013724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:48.019925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:48.020551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:48.020829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:48.021103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:48.022195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:48.023175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463df2d-dfdf-495d-9399-7247defd31ba None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:48.023677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:48.025229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 21/43] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:48.025962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:48.059376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:48.060047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:48.060335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:48.060624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] PATH_INFO: `/v3/domains/default/users/f2b9f22768104befbb8f56575b4c786b/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:48.062664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=f2b9f22768104befbb8f56575b4c786b, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:48.064152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:48.078118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 23/44] 10.222.0.20 () {58 vars in 1361 bytes} [Fri Sep 9 04:24:47 2022] PUT /identity/v3/domains/default/users/f2b9f22768104befbb8f56575b4c786b/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:24:48.079093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:48.140315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45560c79-28bf-43a0-be56-e4d30e5fe15d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:48.181357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c746a-b813-40c9-85a1-7dbd19055a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:48.181998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c746a-b813-40c9-85a1-7dbd19055a44 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:48.182701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c746a-b813-40c9-85a1-7dbd19055a44 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:48.183015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c746a-b813-40c9-85a1-7dbd19055a44 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:48.184157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8c746a-b813-40c9-85a1-7dbd19055a44 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:48.185583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8c746a-b813-40c9-85a1-7dbd19055a44 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:48.204072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b50ad-6909-466b-89d6-59e27915cb8d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:48.216768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:48.251110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac89c4b-8fc1-4e4f-89c8-0a1d3520565f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:48.251637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac89c4b-8fc1-4e4f-89c8-0a1d3520565f None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:48.251919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac89c4b-8fc1-4e4f-89c8-0a1d3520565f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:48.252195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac89c4b-8fc1-4e4f-89c8-0a1d3520565f None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:48.252856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac89c4b-8fc1-4e4f-89c8-0a1d3520565f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:48.254303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac89c4b-8fc1-4e4f-89c8-0a1d3520565f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:48.287883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 24/45] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:24:48 2022] POST /identity/v3/projects => generated 312 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:24:48.316931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 22/46] 10.222.0.20 () {60 vars in 1131 bytes} [Fri Sep 9 04:24:48 2022] POST /identity/v3/roles => generated 223 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:24:50.238954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e89a71-5e6b-4fee-9d56-20d98f37237a None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.239474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e89a71-5e6b-4fee-9d56-20d98f37237a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.239590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e89a71-5e6b-4fee-9d56-20d98f37237a None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.240873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 25/47] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:50.246327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e15861-0282-409d-9497-be0cace20cbc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.246745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e15861-0282-409d-9497-be0cace20cbc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.247035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e15861-0282-409d-9497-be0cace20cbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.266917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-35e15861-0282-409d-9497-be0cace20cbc None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:50.268450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-35e15861-0282-409d-9497-be0cace20cbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:50.322418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:50.334671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 23/48] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:50.464719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16750df7-db34-496c-aaf7-4caf28bd98fe None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.465116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16750df7-db34-496c-aaf7-4caf28bd98fe None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.465476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16750df7-db34-496c-aaf7-4caf28bd98fe None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.467016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 26/49] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:50.472726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.473012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.473288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.491109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:50.492230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:50.561278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 24/50] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:50.598875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16750df7-db34-496c-aaf7-4caf28bd98fe None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:50.696815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16750df7-db34-496c-aaf7-4caf28bd98fe None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:50.729592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:50.730211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.730492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.730769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.741577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:50.747778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:50.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:24:50.749386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 27/51] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:24:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:24:50.754781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:50.828575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:50.852514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:50.854326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc024a8-f98c-4c56-af67-068b1a7f2dfd None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:50.875279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:50.889603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:50.890803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.891130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.891409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.892080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:50.892387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:50.893930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:50.901975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 25/52] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:24:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:50.907657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:50.927402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b2d02a-6a8b-40da-8df9-11892435269d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:50.962066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:50.962690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:50.962996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:50.963281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:50.964396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:50.965888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:50.967484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 28/53] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:50.973359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:51.004517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e815676-7713-4b7c-bf91-09ca173108c6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:51.037166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:51.037802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:51.038087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:51.038367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:51.039564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:51.041021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:51.042598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 26/54] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:24:50 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:51.047961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:51.064268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:51.076083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31cf00-5261-42e2-a669-8d07d42234d7 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:51.100707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:51.116135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:51.116815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:51.117091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:51.117372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:51.118025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:51.119517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:51.146622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 29/55] 10.222.0.20 () {60 vars in 1140 bytes} [Fri Sep 9 04:24:50 2022] POST /identity/v3/projects => generated 298 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:24:51.156612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d1146-c54d-4c93-8113-91ff5130b1e7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:51.193787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ad8d82-6783-4fc9-924b-9de44813d254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:51.194443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ad8d82-6783-4fc9-924b-9de44813d254 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:51.194736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ad8d82-6783-4fc9-924b-9de44813d254 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:51.195053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ad8d82-6783-4fc9-924b-9de44813d254 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:51.196271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ad8d82-6783-4fc9-924b-9de44813d254 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:51.197824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ad8d82-6783-4fc9-924b-9de44813d254 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:51.199556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 27/56] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:24:51.205975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:51.316682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc6a317-6a38-49cf-a321-56ec6604f8ec None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:51.349561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:51.350176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:51.350458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:51.350738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:51.351507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.domain.id=default, user.id=f2b9f22768104befbb8f56575b4c786b) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:51.351817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:51.353254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ca85f4-db8c-45a6-8c46-e30a6926a78c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:51.359478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 30/57] 10.222.0.20 () {56 vars in 1322 bytes} [Fri Sep 9 04:24:51 2022] GET /identity/v3/role_assignments?user.id=f2b9f22768104befbb8f56575b4c786b&scope.domain.id=default&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 530 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:53.285248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf99fdb-cff5-4899-8f48-6a87171f626d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:53.285678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf99fdb-cff5-4899-8f48-6a87171f626d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:53.285810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf99fdb-cff5-4899-8f48-6a87171f626d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:53.287206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 28/58] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:53.292354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:53.292781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:53.292916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:53.306097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:53.312134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:53.313329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:53.376771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 31/59] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:53.648545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf99fdb-cff5-4899-8f48-6a87171f626d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:53.748320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf99fdb-cff5-4899-8f48-6a87171f626d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:53.764395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:53.782053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0311f942-7e05-4f74-bb56-6ec7ff97196d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:53.782668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0311f942-7e05-4f74-bb56-6ec7ff97196d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:53.782973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0311f942-7e05-4f74-bb56-6ec7ff97196d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:53.783254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0311f942-7e05-4f74-bb56-6ec7ff97196d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:53.784347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0311f942-7e05-4f74-bb56-6ec7ff97196d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:53.785780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0311f942-7e05-4f74-bb56-6ec7ff97196d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:53.787345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 29/60] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:53.792769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:53.888145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca667c5-1399-4827-abeb-ca5a1b00b15d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:53.920759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fe6a23-deca-4ff9-80bf-e39b3fd8cc2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:53.921392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe6a23-deca-4ff9-80bf-e39b3fd8cc2c None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:53.921675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe6a23-deca-4ff9-80bf-e39b3fd8cc2c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:53.921957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe6a23-deca-4ff9-80bf-e39b3fd8cc2c None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:53.922617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fe6a23-deca-4ff9-80bf-e39b3fd8cc2c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:53.924089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fe6a23-deca-4ff9-80bf-e39b3fd8cc2c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:53.936018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:53.952960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 32/61] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:24:53 2022] POST /identity/v3/users => generated 275 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:24:57.190805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d0e8f-6950-451d-aeab-a9f5d37d4e50 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:57.191855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d0e8f-6950-451d-aeab-a9f5d37d4e50 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:57.192339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d0e8f-6950-451d-aeab-a9f5d37d4e50 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:57.193918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 30/62] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:57.199496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8463b9-375b-4a73-833d-0b592662b879 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:57.200388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8463b9-375b-4a73-833d-0b592662b879 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:57.200898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8463b9-375b-4a73-833d-0b592662b879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:57.224049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2c8463b9-375b-4a73-833d-0b592662b879 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:57.225562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2c8463b9-375b-4a73-833d-0b592662b879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:57.297502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 33/63] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:57.411524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6c470d-6b9e-4d4a-83b1-4fa23882502c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:57.411842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6c470d-6b9e-4d4a-83b1-4fa23882502c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:57.412082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6c470d-6b9e-4d4a-83b1-4fa23882502c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:57.413665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 31/64] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:57.419029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067381f-1407-4af6-8cbb-21c8b8bc1f6b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:57.419324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067381f-1407-4af6-8cbb-21c8b8bc1f6b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:57.419601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067381f-1407-4af6-8cbb-21c8b8bc1f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:57.439811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a067381f-1407-4af6-8cbb-21c8b8bc1f6b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:57.441066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a067381f-1407-4af6-8cbb-21c8b8bc1f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:57.516719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 34/65] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:57.584454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a3a3e0-6b66-423b-bc10-f7ea81b6114f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:57.584756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a3a3e0-6b66-423b-bc10-f7ea81b6114f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:57.585020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a3a3e0-6b66-423b-bc10-f7ea81b6114f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:57.586362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 32/66] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:57.591710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:57.592003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:57.592311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:57.615284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:57.616522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:57.664960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:57.695390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 35/67] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:57.718482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a3a3e0-6b66-423b-bc10-f7ea81b6114f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:57.847608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:57.980128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dd992c-d7a0-454e-90f7-8d870faef6db None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.001985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a3a3e0-6b66-423b-bc10-f7ea81b6114f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.019881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.020647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.021106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.021389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.033489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:58.038500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:58.039392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:24:58.040141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 36/68] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:24:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:24:58.040751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.041539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f057cd-2642-4c3b-92d3-8ebe053a3ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.041837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f057cd-2642-4c3b-92d3-8ebe053a3ea7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.042124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f057cd-2642-4c3b-92d3-8ebe053a3ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.042404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f057cd-2642-4c3b-92d3-8ebe053a3ea7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.056616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f057cd-2642-4c3b-92d3-8ebe053a3ea7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e4f057cd-2642-4c3b-92d3-8ebe053a3ea7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:58.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:58.062674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:24:58.063771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:24:58.067067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 33/69] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:24:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:24:58.069802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee1ea7-8511-466a-a6ab-f5b4c10a7e70 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.070380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee1ea7-8511-466a-a6ab-f5b4c10a7e70 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.070672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee1ea7-8511-466a-a6ab-f5b4c10a7e70 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.072329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 34/70] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:58.075649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee1ea7-8511-466a-a6ab-f5b4c10a7e70 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.094447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.128490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.154225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65949b7-9cff-4a4b-9724-3b18a8054fec None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.185088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee1ea7-8511-466a-a6ab-f5b4c10a7e70 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.188254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.188889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.189168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.189444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.199598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:58.205217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:58.206051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:24:58.206971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 37/71] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:24:58.207609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.221571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.222357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.222651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.222920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.223620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:58.223928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.225316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b84777-d061-49dd-b7e0-005ef31ea2fb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.234168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 35/72] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.235343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a314a9-08ee-48ef-9ebb-3fbe92b19244 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.235636 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a314a9-08ee-48ef-9ebb-3fbe92b19244 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.235911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a314a9-08ee-48ef-9ebb-3fbe92b19244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.253805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-82a314a9-08ee-48ef-9ebb-3fbe92b19244 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:58.255061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-82a314a9-08ee-48ef-9ebb-3fbe92b19244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:58.315127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a6be9-a616-4c36-a17d-ecd0ebeaf706 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.316622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.321723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 36/73] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:58.322989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb7e22-97af-420c-8a50-431f908657df None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.323280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb7e22-97af-420c-8a50-431f908657df None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.323558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb7e22-97af-420c-8a50-431f908657df None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 37/74] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:24:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:24:58.325465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb7e22-97af-420c-8a50-431f908657df None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.363875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.364494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.364777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.365057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.365760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:58.366071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.367545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.370412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.382031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 38/75] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.382925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.441178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb7e22-97af-420c-8a50-431f908657df None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.478831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.479476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.479760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.480032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.480705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:58.481009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.482448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68811656-fe7f-4557-893c-f02096e3cccc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.491684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 38/76] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.492767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.493063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.493342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.497824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ff8f59-4db9-408e-945d-9aacc4213b47 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.511950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:24:58.513141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:24:58.534273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.535174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.535631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.536097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.537375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.539031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.540777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 39/77] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.541532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.578788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.588388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 39/78] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:24:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:24:58.589240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.619202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.657292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e9bb6-a0c3-428a-82ba-1cc02d4d82af None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.692280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.693130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.693414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.693690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.695213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fb6d7-783c-403a-9417-58ea4a10cf6b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.707516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.709122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.710742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 40/79] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.711458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.737817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.738647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.739130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.739540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.740816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.742456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.744197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 40/80] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.745090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.806549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.818855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb096d2-3c45-481f-8396-98ad923e19ab None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.850012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:58.854597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0060cd1-5703-4c24-b162-0b73e1156cca None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:58.858560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.859214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.859499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.859774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.890289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:58.890916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:58.891227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:58.891506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:58.892645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.894082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.895682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 41/81] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:58.899090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:58.927663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:58.929211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:58.994851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7906dd94-8343-4ec0-b2b7-7f08fbdee937 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.015139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.025636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 41/82] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:24:59.026446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.029337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.030535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.031071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.031190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.064593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.099258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.100720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.125533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73ee3c4-a255-473e-9599-cf9500f89d17 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.160323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.160930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.161209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.161480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.200649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 42/83] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:24:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:24:59.201565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.218214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.229131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.230612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.269303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.304330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0afc31-16e3-46fd-a872-8eefd2e84e11 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.328640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 42/84] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:24:59.333352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.341794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.342984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.343274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.343552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.353607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:24:59.358975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:24:59.359980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:24:59.360755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 43/85] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:24:59.361464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.421543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.431830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb50002-f40a-4296-aa6c-d356e17ebf92 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.460859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa17e8ed-b6a2-42f9-911a-dbda812731b5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.465518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.466159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.466471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.466753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.468083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.469646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.486799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.490027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 43/86] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:24:59 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:24:59.490868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.509876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.510575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.510856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.511235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.512443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.513943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.516253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 44/87] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:59.521483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.610086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cade03b0-f93c-4e50-9335-a2a549e27860 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.636407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58700b4c-1fef-4e56-8ecd-a06b8944dcc1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.650244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.650902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.651250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.651607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.652888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.654997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.656516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 44/88] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:59.656879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.674325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.681434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.682287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.682822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.683135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.684570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.686009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.688975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 45/89] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:59.689992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.766487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.782482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe4960e-58ad-4151-8182-f0b76223ed62 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.821599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.822213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.822496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.822775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.822869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b695169d-2827-4b22-9206-af94d4853dd2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.823479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:59.823785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.825354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.834849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 45/90] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:59.835782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.866766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.867419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.867702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.867974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.868699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=f2b9f22768104befbb8f56575b4c786b) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:59.869011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.870452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:24:59.882111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 46/91] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/role_assignments?user.id=f2b9f22768104befbb8f56575b4c786b&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:24:59.883024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:24:59.931252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:24:59.958276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8be52f8-0677-491f-9ba5-e3712a31b578 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:24:59.994310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:24:59.994920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:24:59.995245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:24:59.995506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:24:59.996252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=9eb2801446984bbe9e1f2c28ec34e108) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:24:59.996753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:24:59.998196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:00.004935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 46/92] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/role_assignments?user.id=9eb2801446984bbe9e1f2c28ec34e108&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=807749e5ca90423999feeb8a7727599c => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:00.005871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:00.018193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32407336-dbd8-4c47-b192-9d1e302ba8c5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:00.055746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:00.069553 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:00.070539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:00.071006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:00.071529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:00.072662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] RBAC: Adding query filter params (role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=9eb2801446984bbe9e1f2c28ec34e108) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:00.073129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:00.074820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:00.083192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 47/93] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:24:59 2022] GET /identity/v3/role_assignments?user.id=9eb2801446984bbe9e1f2c28ec34e108&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:00.150407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799edeb4-9515-46ca-98b2-5b3d7149d3fd None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:00.200976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:00.201652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:00.202470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:00.202746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:00.204936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:00.206837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:00.209247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 47/94] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:00 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:00.214807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:00.352751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b92e13-c8f4-443c-b461-9d4d2b45d42c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:00.388915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:00.390226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:00.390603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:00.391005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:00.452326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:00.462971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:00.464712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:00.577512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 48/95] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:00.584090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:00.606737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:00.751574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597196c4-1ac5-4658-b19e-267817dd77bb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:00.796838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7863c27d-3640-43a4-89f2-c43fad27a2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:00.797480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7863c27d-3640-43a4-89f2-c43fad27a2fa None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:00.797768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7863c27d-3640-43a4-89f2-c43fad27a2fa None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:00.798043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7863c27d-3640-43a4-89f2-c43fad27a2fa None admin] PATH_INFO: `/v3/projects/4186528919c84f3f8fa8e21068a3740b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:00.799227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7863c27d-3640-43a4-89f2-c43fad27a2fa None admin] RBAC: Authorizing `identity:get_project(project_id=4186528919c84f3f8fa8e21068a3740b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:00.800682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7863c27d-3640-43a4-89f2-c43fad27a2fa None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:00.802472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 48/96] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:00 2022] GET /identity/v3/projects/4186528919c84f3f8fa8e21068a3740b => generated 312 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:00.808752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:00.925371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:00.935742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5afcc2-8a0b-45be-8f9d-fa30f1701ebf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:00.971735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:00.972488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:00.972869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:00.973243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:00.974068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=4186528919c84f3f8fa8e21068a3740b, user.id=9eb2801446984bbe9e1f2c28ec34e108) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:00.974478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:00.976060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e27050-56c3-4d2d-9d67-57a24b4fd0a3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:00.982242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 49/97] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:00 2022] GET /identity/v3/role_assignments?user.id=9eb2801446984bbe9e1f2c28ec34e108&scope.project.id=4186528919c84f3f8fa8e21068a3740b&role.id=807749e5ca90423999feeb8a7727599c => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:02.913250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba8583-d5d0-4595-a1ea-f5ff9a8f0dec None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:02.913764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba8583-d5d0-4595-a1ea-f5ff9a8f0dec None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:02.913840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba8583-d5d0-4595-a1ea-f5ff9a8f0dec None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:02.915329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 49/98] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:02.920674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82584483-430a-4648-87ce-e3fd4d9e0e88 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:02.921156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82584483-430a-4648-87ce-e3fd4d9e0e88 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:02.921236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82584483-430a-4648-87ce-e3fd4d9e0e88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:02.952041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-82584483-430a-4648-87ce-e3fd4d9e0e88 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:02.953248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-82584483-430a-4648-87ce-e3fd4d9e0e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:03.017731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594bef74-72d0-43a2-8768-7b825763fc5c None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.018065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594bef74-72d0-43a2-8768-7b825763fc5c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.018291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594bef74-72d0-43a2-8768-7b825763fc5c None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.019798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 50/99] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:03.025745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3491a-0ca5-4932-8866-b97e703eebcb None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.026047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3491a-0ca5-4932-8866-b97e703eebcb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.026315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3491a-0ca5-4932-8866-b97e703eebcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.037684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 50/100] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:03.072053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0ec3491a-0ca5-4932-8866-b97e703eebcb None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:03.073286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0ec3491a-0ca5-4932-8866-b97e703eebcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:03.163818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 51/101] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 09 04:25:03.302363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844879a7-1abb-4d3b-87c2-acc75354da00 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.302664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844879a7-1abb-4d3b-87c2-acc75354da00 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.302942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844879a7-1abb-4d3b-87c2-acc75354da00 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.304421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 51/102] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:03.320235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.320549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.320809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.337112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:03.347544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:03.348781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:03.369997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46edc1b1-27c3-4bff-a377-7f847a24f2ed None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.370315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46edc1b1-27c3-4bff-a377-7f847a24f2ed None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.370561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46edc1b1-27c3-4bff-a377-7f847a24f2ed None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.371955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 52/103] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:03.380485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b6013-fed9-4489-8169-574a98a8249b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.380772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b6013-fed9-4489-8169-574a98a8249b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.381053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b6013-fed9-4489-8169-574a98a8249b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.405288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-960b6013-fed9-4489-8169-574a98a8249b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:03.406494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-960b6013-fed9-4489-8169-574a98a8249b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:03.476042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 52/104] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:03.476645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:03.543549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 53/105] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:03.546518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d80e1a-fd3d-44bc-815a-598d8e08abeb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.547202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d80e1a-fd3d-44bc-815a-598d8e08abeb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.547691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d80e1a-fd3d-44bc-815a-598d8e08abeb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.549364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 54/106] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:03.555121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.555644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.556108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.577436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:03.578977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:03.589337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:03.671678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea6d72a-1a8c-4702-ad96-55697a978274 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:03.709421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 55/107] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:03.710854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:03.714853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:03.715528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.715814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.716090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.727333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:03.741427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:03.750250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:03.751292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:03.752228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:03.752816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 53/108] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:03.759971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:03.868097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1433721b-fc3b-4be8-8417-8c4bffcf4bde None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:03.916957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:03.917913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.918327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.918741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.932145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:03.939435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:03.940468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:03.941400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:03.942672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c32ddf5-78bc-4306-bc28-e19c3c51d7a7 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:03.943801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 56/109] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:03.944861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:03.949577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:03.984533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:03.985217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:03.985604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:03.986004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:03.987021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:03.987382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:03.988895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.002277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 54/110] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.002277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.080730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.082550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be0807-7fc4-4b02-b61c-303d851c8371 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.117997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.118686 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.118959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.119236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.126378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727c8444-f138-4bda-a23f-6d619551540b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.129980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:04.134909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:04.135751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 57/111] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:04.141783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.169476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.170103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.170924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.171226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.172589 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.174038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.175628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 55/112] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.179800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.200547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.238212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314fcbc-52e4-4c6d-8e99-29dbac8b6f27 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.275448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.276059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.276342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.276638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.276964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a22fea-5c64-43b1-b25d-ea1f3d8c5b83 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.277286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:04.277592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.279010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.287429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 58/113] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.288213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.302372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.312348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.312979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.313260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.313536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.315109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.316581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.318140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 56/114] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.318885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.389741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9d5d59-aecd-4bf5-81cb-a369c5ac5edb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.414156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecb2b24-a5cb-4865-a087-c363911650ca None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.415070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.424046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1e1055-0eb1-4958-8cb9-37f98a3a3c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.424673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1e1055-0eb1-4958-8cb9-37f98a3a3c04 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.424965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1e1055-0eb1-4958-8cb9-37f98a3a3c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.425242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1e1055-0eb1-4958-8cb9-37f98a3a3c04 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.435551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1e1055-0eb1-4958-8cb9-37f98a3a3c04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0d1e1055-0eb1-4958-8cb9-37f98a3a3c04 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:04.441316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:04.442171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:04.443111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 59/115] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:04.445493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06df76-fe18-423a-bc8f-ad81e1d247ce None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.445851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06df76-fe18-423a-bc8f-ad81e1d247ce None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.446140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06df76-fe18-423a-bc8f-ad81e1d247ce None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.447536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 60/116] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:04.448408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.449013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.449296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.449570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.450268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:04.450577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.451776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06df76-fe18-423a-bc8f-ad81e1d247ce None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.452112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.460717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 57/117] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.461404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.515590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.553193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06df76-fe18-423a-bc8f-ad81e1d247ce None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.557894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3c011-9546-404d-8ac1-c6ce9af2352a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.590312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.591418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.591569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.591937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.592753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.593101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.593195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.593273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] PATH_INFO: `/v3/users/cb200a44e3e34c12b3c405c33d703907` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.593602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.596396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.608902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] RBAC: Authorizing `identity:get_user(user_id=cb200a44e3e34c12b3c405c33d703907)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.611286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.618281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 58/118] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/users/cb200a44e3e34c12b3c405c33d703907 => generated 289 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.618281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.632261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 61/119] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:25:04 2022] POST /identity/v3/projects => generated 302 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:04.633298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.692603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.730912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15b7415-193c-4f95-8796-8295c0bfaa13 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.747275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.766128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b389fab4-fdeb-4095-b0fe-c56035c6aeef None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.773494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff8412e-ce3b-4afa-9e6b-477d166f8620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.774162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff8412e-ce3b-4afa-9e6b-477d166f8620 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.774447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff8412e-ce3b-4afa-9e6b-477d166f8620 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.774718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff8412e-ce3b-4afa-9e6b-477d166f8620 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.776577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff8412e-ce3b-4afa-9e6b-477d166f8620 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.778526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff8412e-ce3b-4afa-9e6b-477d166f8620 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.785863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 59/120] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.785863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.785863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.785863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.805703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.806385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.806672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.806979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.807567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:04.807694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:04.807943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.809341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:04.809582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:04.819892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 62/121] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:04.823035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.883946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 60/122] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:04.884640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:04.942119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:04.947579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dfef22-f99a-4a3b-9485-efabf5b9b9b5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.992325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc374b34-18e3-4449-99c4-d34a8b920d83 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:04.993230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:04.994235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:04.995534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:04.996006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:04.997394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:04.999111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.001361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 63/123] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:05.002280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.033605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.034548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.034704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.035138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.039280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.109283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.111145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.130960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16b01b6-50e4-48b4-9e12-90573228e61d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.177509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.178341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.178638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.178923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/f2b9f22768104befbb8f56575b4c786b/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.180943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb4febd431cc435cbade69ba4716e3ec, user_id=f2b9f22768104befbb8f56575b4c786b, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.182423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.202983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 64/124] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:05 2022] PUT /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/f2b9f22768104befbb8f56575b4c786b/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:05.202983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.226713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 61/125] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:05.228377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.258806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.266410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.395795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ebe3ab-5a68-42c8-9326-23d92a368815 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.436422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e046a28-ba48-49f8-94fb-d8f9227dce3a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.440934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.441587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.441870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.442148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.443539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.445461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.447481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 65/126] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:05.448257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.476418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.477448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.477740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.478296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.479693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.481767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.484224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 62/127] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:05.485013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.567951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.572784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5153e5-584b-4ca3-9da3-d0b6cac4177d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.595324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.614277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.614916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.615228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.615506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.620661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfefd46-6bfc-488b-bd26-1199f1d8cb8d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.631794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:05.639425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:05.640478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:05.641439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:05.644090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 66/128] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:05.645014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.661188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.661815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.662168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.662592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.663740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.665267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.672648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 63/129] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:05.673546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.780485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b75a6-9ec3-4f02-a9d5-4f831a0e4b36 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.783285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa661a3f-ab72-4461-b617-2a1ca851ddfc None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:05.820310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.821204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.821661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.822087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.823406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.825012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.826734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:05.827261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 67/130] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:05.827497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:05.827792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:05.828067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/9eb2801446984bbe9e1f2c28ec34e108/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:05.828672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.830256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb4febd431cc435cbade69ba4716e3ec, user_id=9eb2801446984bbe9e1f2c28ec34e108, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:05.832195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:05.853424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 64/131] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:05 2022] PUT /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/9eb2801446984bbe9e1f2c28ec34e108/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:05.854814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:05.871028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.871276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:05.987228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bd0f4d-b574-4a57-b4a0-a4558f425577 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:06.029033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675d9c08-8a47-4660-8eb1-239d00b0d2a8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:06.030148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:06.030692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:06.031008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:06.031296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:06.032039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] RBAC: Adding query filter params (role.id=55face8aaf314343b721a50b5cabf886, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=cb200a44e3e34c12b3c405c33d703907) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:06.032358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:06.037633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:06.041545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 65/132] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/role_assignments?user.id=cb200a44e3e34c12b3c405c33d703907&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=55face8aaf314343b721a50b5cabf886 => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:06.042304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:06.093260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:06.094260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:06.094757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:06.095289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:06.096250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:06.096775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:06.098449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:06.115472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 68/133] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:06.133930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:06.141658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:06.179551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02951a9-2bca-491e-a532-7ab8b01c84a0 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:06.222730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:06.223704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:06.223998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:06.224437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/9eb2801446984bbe9e1f2c28ec34e108/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:06.227043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb4febd431cc435cbade69ba4716e3ec, user_id=9eb2801446984bbe9e1f2c28ec34e108, role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:06.228998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:06.256121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 66/134] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:06 2022] PUT /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/9eb2801446984bbe9e1f2c28ec34e108/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:06.292445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:06.312750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5f0b11-cf22-408e-9206-38543283114e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:06.367906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:06.368953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:06.369442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:06.369975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:06.375856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:06.377676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:06.380162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 69/135] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:06 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:06.388154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:06.539883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b0e624-8c73-465a-936d-14504f3b5324 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:06.578377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:06.584114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ffe340-826b-44e6-83d0-f5a18915ea39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:06.584757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ffe340-826b-44e6-83d0-f5a18915ea39 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:06.585099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ffe340-826b-44e6-83d0-f5a18915ea39 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:06.585403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ffe340-826b-44e6-83d0-f5a18915ea39 None admin] PATH_INFO: `/v3/projects/4186528919c84f3f8fa8e21068a3740b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:06.588293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ffe340-826b-44e6-83d0-f5a18915ea39 None admin] RBAC: Authorizing `identity:get_project(project_id=4186528919c84f3f8fa8e21068a3740b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:06.589770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ffe340-826b-44e6-83d0-f5a18915ea39 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:06.591486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 67/136] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:06 2022] GET /identity/v3/projects/4186528919c84f3f8fa8e21068a3740b => generated 312 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:06.599011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:06.738696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6fdda3-02e7-4dfb-89d3-2fa57704b950 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:06.783787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7de53-89df-4759-9aa0-21a8d2942ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:06.784461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7de53-89df-4759-9aa0-21a8d2942ca4 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:06.784746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7de53-89df-4759-9aa0-21a8d2942ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:06.785022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7de53-89df-4759-9aa0-21a8d2942ca4 None admin] PATH_INFO: `/v3/projects/4186528919c84f3f8fa8e21068a3740b/users/9eb2801446984bbe9e1f2c28ec34e108/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:06.786959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7de53-89df-4759-9aa0-21a8d2942ca4 None admin] RBAC: Authorizing `identity:create_grant(project_id=4186528919c84f3f8fa8e21068a3740b, user_id=9eb2801446984bbe9e1f2c28ec34e108, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:06.788408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7de53-89df-4759-9aa0-21a8d2942ca4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:06.816300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 70/137] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:06 2022] PUT /identity/v3/projects/4186528919c84f3f8fa8e21068a3740b/users/9eb2801446984bbe9e1f2c28ec34e108/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:07.205776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea5221-13ab-490c-a24f-c0543c9a1fb4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:07.206291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea5221-13ab-490c-a24f-c0543c9a1fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:07.206391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea5221-13ab-490c-a24f-c0543c9a1fb4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:07.207828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 68/138] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:07.213560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:07.213977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:07.214160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:07.248447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:07.249669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:07.264727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:07.348813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 71/139] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:07.896975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea5221-13ab-490c-a24f-c0543c9a1fb4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:08.030857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea5221-13ab-490c-a24f-c0543c9a1fb4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:08.077041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ade1a4-e68a-406e-b41b-27884bb1d61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:08.077683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ade1a4-e68a-406e-b41b-27884bb1d61b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:08.077965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ade1a4-e68a-406e-b41b-27884bb1d61b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:08.078241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ade1a4-e68a-406e-b41b-27884bb1d61b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:08.079399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ade1a4-e68a-406e-b41b-27884bb1d61b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:08.080853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ade1a4-e68a-406e-b41b-27884bb1d61b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:08.082583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 69/140] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:08.088316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:08.127929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a48eb9-d085-4dde-8712-b1f8d7b226b7 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:08.128273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a48eb9-d085-4dde-8712-b1f8d7b226b7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:08.128490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a48eb9-d085-4dde-8712-b1f8d7b226b7 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:08.129960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 70/141] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:08.134683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:08.135000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:08.135285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:08.160617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:08.161808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:08.260070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a06477-c05f-4a45-8545-4b8e2a8a253b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:08.291594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:08.300826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 71/142] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:08.319719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:08.320346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:08.320638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:08.320919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:08.321606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:08.323075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:08.361720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:08.366302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 72/143] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:08 2022] POST /identity/v3/users => generated 283 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:08.717048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:08.827671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd3673d-da73-435e-91b6-9135cecd2f9a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:08.875178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:08.876236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:08.876728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:08.877216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:08.889536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:08.894769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:08.895702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:08.896496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:08.898726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 72/144] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:08.905430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:09.037104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b548506-d95a-4809-ad87-7cf55c2e7c77 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:09.078546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:09.079205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.079501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.079778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.080461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:09.080768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:09.082158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03db88c0-cf9f-48ba-8b7c-8dcd8f41c61b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:09.091273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 73/145] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:09.097069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:09.184021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:09.246443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809f7af-b338-480c-bbee-cebf586f5260 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:09.280584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3803bb5a-03d6-4e7d-b4af-58cfd9da7ac0 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.281469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3803bb5a-03d6-4e7d-b4af-58cfd9da7ac0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.281927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3803bb5a-03d6-4e7d-b4af-58cfd9da7ac0 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.284092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 74/146] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:09.289729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19809a2b-e1b6-45b4-b013-f3bbd0b1b7e6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.290218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19809a2b-e1b6-45b4-b013-f3bbd0b1b7e6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.290655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19809a2b-e1b6-45b4-b013-f3bbd0b1b7e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.298759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:09.299515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.299802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.300077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] PATH_INFO: `/v3/users/f2b9f22768104befbb8f56575b4c786b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.302018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] RBAC: Authorizing `identity:get_user(user_id=f2b9f22768104befbb8f56575b4c786b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:09.304811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:09.305536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 73/147] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:09 2022] GET /identity/v3/users/f2b9f22768104befbb8f56575b4c786b => generated 247 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:09.313318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:09.322643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-19809a2b-e1b6-45b4-b013-f3bbd0b1b7e6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:09.324275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19809a2b-e1b6-45b4-b013-f3bbd0b1b7e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:09.421685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:09.433991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 75/148] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:09.439417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62863754-2955-49a1-9fc4-34767ec9bd8d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.439721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62863754-2955-49a1-9fc4-34767ec9bd8d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.440003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62863754-2955-49a1-9fc4-34767ec9bd8d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.441380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 76/149] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:09 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:09.446832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa57d69a-7429-479f-b240-390e3734663e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.447157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa57d69a-7429-479f-b240-390e3734663e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.447432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa57d69a-7429-479f-b240-390e3734663e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.450760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a954967-2361-4b39-9c11-4c093ab6e4b6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:09.472460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fa57d69a-7429-479f-b240-390e3734663e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:09.473696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fa57d69a-7429-479f-b240-390e3734663e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:09.486527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:09.500613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:09.501590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.502057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.502520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.593584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 77/150] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:09.595729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f475069d-cda8-4140-b077-9cc4ac74da58 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.596025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f475069d-cda8-4140-b077-9cc4ac74da58 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.596303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f475069d-cda8-4140-b077-9cc4ac74da58 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.597558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 78/151] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:09.603643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec30e040-a736-4184-a962-243496c4c750 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:09.603930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec30e040-a736-4184-a962-243496c4c750 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:09.604205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec30e040-a736-4184-a962-243496c4c750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:09.644081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ec30e040-a736-4184-a962-243496c4c750 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:09.645302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ec30e040-a736-4184-a962-243496c4c750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:09.648212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:09.649742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:09.788168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 79/152] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:09.817912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 74/153] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:09.824269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec30e040-a736-4184-a962-243496c4c750 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:09.900349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:09.960069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec30e040-a736-4184-a962-243496c4c750 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:09.990048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.002498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.003251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.003538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.003815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.005046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:10.006588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:10.008428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 80/154] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:09 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:10.030847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.040906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:10.140684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94406ba9-8b38-409a-a8c1-d8bed94226a1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.193277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.193837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.194116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.194376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.205549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:10.210511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:10.211392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:10.212143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 75/155] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:10.212778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.245892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fbbb0-9e64-4e44-b176-5637b74c18c2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.273489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:10.292819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.293465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.293694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.293961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.294697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=f2b9f22768104befbb8f56575b4c786b) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:10.295066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:10.296735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:10.303857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 81/156] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:10 2022] GET /identity/v3/role_assignments?user.id=f2b9f22768104befbb8f56575b4c786b&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:10.304490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.347685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:10.394114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd499f7-3ce3-4262-9ae0-c36f4f4d71eb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.485621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec4c9cc-f4fc-4065-9363-8e820c44d26e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.506959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.507637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.507946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.508193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.537362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:10.550828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:10.552282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:10.553132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 76/157] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:10.553777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.579665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e5e332-9fdd-48fc-b6c7-88706bae80c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.580325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5e332-9fdd-48fc-b6c7-88706bae80c0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.580612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5e332-9fdd-48fc-b6c7-88706bae80c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.580897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5e332-9fdd-48fc-b6c7-88706bae80c0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.602500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e5e332-9fdd-48fc-b6c7-88706bae80c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a7e5e332-9fdd-48fc-b6c7-88706bae80c0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:10.609106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:10.609925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:10.610768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:10.618811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 82/158] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:10.628452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246f4a9f-cde9-4425-88ee-4b568bde4806 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.628728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246f4a9f-cde9-4425-88ee-4b568bde4806 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.629535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246f4a9f-cde9-4425-88ee-4b568bde4806 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.630587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 83/159] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:10.631229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246f4a9f-cde9-4425-88ee-4b568bde4806 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.700982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3ada4-5c55-4ddf-8d81-1d38bec6003e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.742142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.743125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.743622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.744113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.745481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:10.746060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:10.748392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:10.753007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:10.760045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 77/160] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:10.761244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:10.775413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:10.853595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246f4a9f-cde9-4425-88ee-4b568bde4806 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.918518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.919545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.920562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.922383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.923286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:10.924268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:10.926209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4fbeeb-116d-424c-9bb6-24d7fb550ac3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:10.939213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 84/161] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:10.940445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c9ea7c-4ffc-41df-97f0-2a736c3e83b2 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:10.941027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.941581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.942062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.976054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:10.977578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:10.991438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:10.992240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:10.992482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:10.992727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:10.993398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:10.993701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:10.995108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:11.012146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 78/162] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:11.013076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:11.112669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 85/163] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:11.117248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:11.144268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:11.198956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed4048b-86da-4edb-a093-eb5d70bb7542 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:11.265568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:11.265845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:11.266149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:11.266546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:11.267575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:11.269154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:11.270737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 79/164] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:11 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:11.277836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:11.279031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:11.429264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d49087-d740-4d97-adf4-441d06dd4332 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:11.510183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:11.510850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:11.512042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:11.512042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:11.512582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:11.514560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:11.524510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 86/165] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:11 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 411 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:11.525672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:11.542142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff61538-3122-41a8-9784-74a8e20d67d8 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:11.601139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:11.602124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:11.603607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:11.604081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] PATH_INFO: `/v3/users/cb200a44e3e34c12b3c405c33d703907` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:11.606081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] RBAC: Authorizing `identity:get_user(user_id=cb200a44e3e34c12b3c405c33d703907)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:11.613606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:11.615492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 80/166] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:11 2022] GET /identity/v3/users/cb200a44e3e34c12b3c405c33d703907 => generated 289 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:11.616444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:11.636609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:11.746300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9ee9c7-5935-46c3-8b5d-6b9aecdd958d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:11.750751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:11.817050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:11.819125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:11.819795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:11.821194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:11.854895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377fab9c-65f8-4edf-a6e2-2ee2f0563093 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:11.934500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:11.935165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:11.936118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:11.936401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:11.978061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:11.979586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.070883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.072948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.081323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.137487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 87/167] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 612 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:12.138983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.142532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.184566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 81/168] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 568 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:12.190085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.267761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5558ff-e22b-4a0c-9639-0c03bf958549 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.304189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b4286-c1ac-4837-ab87-6eb4947daff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.305119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b4286-c1ac-4837-ab87-6eb4947daff9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.305601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b4286-c1ac-4837-ab87-6eb4947daff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.306066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b4286-c1ac-4837-ab87-6eb4947daff9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.317170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799b4286-c1ac-4837-ab87-6eb4947daff9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:12.320424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803ca879-0334-4598-9629-3979f5c6f6a4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-799b4286-c1ac-4837-ab87-6eb4947daff9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:12.322690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:12.323702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:12.324474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:12.327211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 88/169] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:12.328592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed710ae2-627b-4422-b263-161ac1c5f0a1 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.328905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.329659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed710ae2-627b-4422-b263-161ac1c5f0a1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.330135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed710ae2-627b-4422-b263-161ac1c5f0a1 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.332050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 89/170] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:12.335954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed710ae2-627b-4422-b263-161ac1c5f0a1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.363261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.363902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.364183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.364448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.365760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.367881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.369865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 82/171] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:12.370965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.411919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.455511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed710ae2-627b-4422-b263-161ac1c5f0a1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.492602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.493571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.494064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.494590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.494693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7de50c-5ff8-438f-81aa-2f7095744970 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.496037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.497694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.499856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 90/172] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:12.505548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.531184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70abd481-b30f-481a-a909-0e0f328ee64c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.531828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70abd481-b30f-481a-a909-0e0f328ee64c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.532110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70abd481-b30f-481a-a909-0e0f328ee64c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.532381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70abd481-b30f-481a-a909-0e0f328ee64c None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.533539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70abd481-b30f-481a-a909-0e0f328ee64c None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.534986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70abd481-b30f-481a-a909-0e0f328ee64c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.536635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 83/173] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:12.538040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.538328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.538605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.557775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:12.559234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:12.598034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.621643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b7c4a-6326-4aad-8d82-f12ee8a445a1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.632869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 84/174] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:12.639104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.672324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.680174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.681296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.681773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.682297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.683183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:12.683981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.686230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.705551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 91/175] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:12.706808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.748659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509b12b-ae67-44c0-8af4-196bca6819f3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.786429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.787165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.787453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.787731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/cb200a44e3e34c12b3c405c33d703907/roles/55face8aaf314343b721a50b5cabf886` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.790521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb4febd431cc435cbade69ba4716e3ec, user_id=cb200a44e3e34c12b3c405c33d703907, role_id=55face8aaf314343b721a50b5cabf886)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.792071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.817398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 85/176] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:12 2022] PUT /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec/users/cb200a44e3e34c12b3c405c33d703907/roles/55face8aaf314343b721a50b5cabf886 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:12.818397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:12.865744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:12.915749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02daf81-3eca-43fa-b5e1-c2270887afcb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.956977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:12.957350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60d25c-596c-4851-a6fc-00250cda8483 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:12.957640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:12.957928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:12.958203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:12.959248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=9eb2801446984bbe9e1f2c28ec34e108) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:12.959567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:12.961173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:12.968284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 92/177] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/role_assignments?user.id=9eb2801446984bbe9e1f2c28ec34e108&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=807749e5ca90423999feeb8a7727599c => generated 608 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:12.969286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.006165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.006957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.007363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.007676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.008443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] RBAC: Adding query filter params (role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=9eb2801446984bbe9e1f2c28ec34e108) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:13.008753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:13.010595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edbb7d7-9342-4720-b3fa-36eb4f4c2706 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:13.018900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 86/178] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/role_assignments?user.id=9eb2801446984bbe9e1f2c28ec34e108&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:13.020611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aba080-fb96-4225-9a9d-646c670d83a4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.020892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aba080-fb96-4225-9a9d-646c670d83a4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.021169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aba080-fb96-4225-9a9d-646c670d83a4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.022809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 87/179] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:13.024047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf97ea-dca5-43ec-83ca-638c84363269 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.024397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf97ea-dca5-43ec-83ca-638c84363269 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.024643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf97ea-dca5-43ec-83ca-638c84363269 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.026107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 88/180] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:13.032354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f8eabe-680b-40ea-b63c-0c42c9f17a61 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.032668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f8eabe-680b-40ea-b63c-0c42c9f17a61 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.032965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f8eabe-680b-40ea-b63c-0c42c9f17a61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.055138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-33f8eabe-680b-40ea-b63c-0c42c9f17a61 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:13.055718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.057971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-33f8eabe-680b-40ea-b63c-0c42c9f17a61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:13.131653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcdd3a0-529e-4600-a12b-13076f285788 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.167490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 89/181] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:13.169182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.170252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.170544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.179763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.181408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.181698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.181986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] PATH_INFO: `/v3/users/9eb2801446984bbe9e1f2c28ec34e108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.183704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] RBAC: Authorizing `identity:get_user(user_id=9eb2801446984bbe9e1f2c28ec34e108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:13.185212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:13.187855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 93/182] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:12 2022] GET /identity/v3/users/9eb2801446984bbe9e1f2c28ec34e108 => generated 275 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:13.189215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.198458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:13.199771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:13.233888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.311292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 90/183] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:13.311985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.350543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d494c3-efd3-4359-b98e-cfefb046ce94 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.392934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.404966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.406220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.406696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.407199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.420209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:13.427429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:13.428306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:13.429076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:13.431506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 94/184] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:13.438417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.444603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5eebe-448b-4d28-87b1-b867b6025066 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.513386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.517453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.517736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.518005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.575308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.630014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d43474f-d5b1-4624-9298-aa4110c9bdd0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.638861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:13.640850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:13.668822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.669422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.669721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.670034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.670707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:13.671113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:13.672798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:13.682061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 95/185] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:13.683186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.739552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.762200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 91/186] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:13.767539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.795391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616946ca-1aef-4074-aaa1-bc437aafac5c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.826522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.830550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.831209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.831481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.831754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.833001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:13.834419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:13.836164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 96/187] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:13.839780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.877671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d0f97-65fe-43f3-965f-0c4dc6ef6578 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.913159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.913790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.914064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.914333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:13.925849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:13.931401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:13.932275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:13.933041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:13.937719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 92/188] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:13.938503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:13.952611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932bc6d9-4753-47d2-abe4-a0f2ce8c0fb1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:13.970717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:13.991911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:13.992830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:13.993111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:13.993384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] PATH_INFO: `/v3/users/b462612c9758449b8a37ab55e52c0570` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.009179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] RBAC: Authorizing `identity:get_user(user_id=b462612c9758449b8a37ab55e52c0570)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.010638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.012330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 97/189] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/users/b462612c9758449b8a37ab55e52c0570 => generated 283 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:14.013089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.051421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c29e29-35af-42d3-be09-5e3919219c30 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.069843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:14.088896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.089828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.090303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.090785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] PATH_INFO: `/v3/projects/4186528919c84f3f8fa8e21068a3740b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.092152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] RBAC: Authorizing `identity:get_project(project_id=4186528919c84f3f8fa8e21068a3740b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.094212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.096741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 93/190] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:13 2022] GET /identity/v3/projects/4186528919c84f3f8fa8e21068a3740b => generated 312 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:14.098007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.121377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452efdf6-7c39-4915-8f5e-eb3e7768c0b7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.161730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.162660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.162974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.163253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.163914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.165362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.191002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 98/191] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:14 2022] POST /identity/v3/users => generated 297 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:14.191898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.216079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79beac77-130d-4b8b-b3c0-dc3060bb49d5 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.241037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:14.259787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.260461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.260743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.261067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.261860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:14.262171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.264001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.273397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 94/192] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:14.274351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.327658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:14.345319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c5a75c-0c61-4f9b-807b-8808e6c20ef3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.384320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.384945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.385232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.385500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.403377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02bbcb6-b089-4a07-9b87-c2c3e9cf1d66 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.441732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.442261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.442691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.442795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.443550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=4186528919c84f3f8fa8e21068a3740b, user.id=9eb2801446984bbe9e1f2c28ec34e108) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:14.443857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.445300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.457240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 95/193] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/role_assignments?user.id=9eb2801446984bbe9e1f2c28ec34e108&scope.project.id=4186528919c84f3f8fa8e21068a3740b&role.id=807749e5ca90423999feeb8a7727599c => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:14.458066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.477516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.479485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.513261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:14.592964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08123a5a-2493-40dc-85fa-f179628a437c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.606905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:14.630062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 99/194] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:14.635152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.644582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.645593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.646018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.646436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] PATH_INFO: `/v3/users/b462612c9758449b8a37ab55e52c0570` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.647850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] RBAC: Authorizing `identity:get_user(user_id=b462612c9758449b8a37ab55e52c0570)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.649543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.651947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 96/195] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/users/b462612c9758449b8a37ab55e52c0570 => generated 283 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:14.658733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:14.809764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d8c58b-b3d5-4fad-b4d5-b7198311bcc2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.854681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:14.868907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193adddb-3251-4559-8a8e-e6a5a20ac7a0 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:14.870049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:14.871961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:14.876515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:14.877018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:14.878681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:14.880314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:14.883713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 100/196] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:14.892287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:15.009514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:15.009514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:15.009514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:15.009514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:15.087658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96846c29-e0e5-4ec9-88dd-de5cdbd6969c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:15.137790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:15.139105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:15.139386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:15.139819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:15.140571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=b462612c9758449b8a37ab55e52c0570) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:15.140882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:15.142333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:15.169214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 101/197] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/role_assignments?user.id=b462612c9758449b8a37ab55e52c0570&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:15.184440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:15.248011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:15.249530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:15.449295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 97/198] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 791 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:15.455923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:15.568809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:15.619946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a41141-f282-4a1c-b8c8-a04ebf47f1fb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:15.657729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca46e6c-0818-4fbf-a465-7847ece458f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:15.659188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca46e6c-0818-4fbf-a465-7847ece458f7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:15.659658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca46e6c-0818-4fbf-a465-7847ece458f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:15.660114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca46e6c-0818-4fbf-a465-7847ece458f7 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:15.661619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca46e6c-0818-4fbf-a465-7847ece458f7 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:15.663398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca46e6c-0818-4fbf-a465-7847ece458f7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:15.665779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 102/199] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:15 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:15.671849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:15.711479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c06e55-b4e2-4c43-b324-70af8d8bf6de None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:15.711665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c06e55-b4e2-4c43-b324-70af8d8bf6de None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:15.711943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c06e55-b4e2-4c43-b324-70af8d8bf6de None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:15.713539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 103/200] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:15.719153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:15.719439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:15.719715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:15.747570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:15.749309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:15.801358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:15.829829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356d8415-9569-439b-9099-b3ffd1890624 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:15.893989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 104/201] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:15.896645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:15.897353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:15.897730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:15.898097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:15.898957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] RBAC: Adding query filter params (role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=b462612c9758449b8a37ab55e52c0570) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:15.899367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:15.900885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:15.907448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 98/202] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:15 2022] GET /identity/v3/role_assignments?user.id=b462612c9758449b8a37ab55e52c0570&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 259 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:16.216216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:16.289290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:16.338842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3bcd24-5d15-44d1-aee3-01c4ceb710fd None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:16.380179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:16.380805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:16.381080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:16.381349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:16.396559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:16.401353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:16.402198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:16.403381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 105/203] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:16.409130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:16.549059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8871e2-c147-4393-b681-74e78bc6ac8d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:16.592725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:16.593334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:16.593613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:16.593880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:16.594683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] RBAC: Adding query filter params (name=reader) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:16.595012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:16.596442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:16.610064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 99/204] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:16.615407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:16.760971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ab35f4-f68f-4ef7-a36e-ff927f7bca14 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:16.791074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:16.810980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:16.811550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:16.811825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:16.812092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] PATH_INFO: `/v3/users/cb200a44e3e34c12b3c405c33d703907` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:16.813250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] RBAC: Authorizing `identity:get_user(user_id=cb200a44e3e34c12b3c405c33d703907)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:16.814705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:16.816597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 106/205] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:16 2022] GET /identity/v3/users/cb200a44e3e34c12b3c405c33d703907 => generated 289 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:16.822072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:16.825993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:16.953991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5bf609-d825-4c6a-b965-89d2c3c573da None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:16.988811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1135581-368b-4c31-b929-5e367d6a0b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:16.989428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1135581-368b-4c31-b929-5e367d6a0b73 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:16.989747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1135581-368b-4c31-b929-5e367d6a0b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:16.990024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1135581-368b-4c31-b929-5e367d6a0b73 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.059635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1135581-368b-4c31-b929-5e367d6a0b73 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:17.061735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1135581-368b-4c31-b929-5e367d6a0b73 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:17.064985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:17.181867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 100/206] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:17.188023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:17.209147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cac670-af15-4737-8504-80fe2f55bbe0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.209439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cac670-af15-4737-8504-80fe2f55bbe0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.209723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cac670-af15-4737-8504-80fe2f55bbe0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.211455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 101/207] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:17.217354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.217644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.217926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.241168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:17.241168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:17.350400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 102/208] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:17.368631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba1d7db-024b-4014-a0b6-ba2b76ac97f4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:17.411783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8bf288-5965-4eb4-9f9d-bf9b505d4aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:17.412689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8bf288-5965-4eb4-9f9d-bf9b505d4aef None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.413137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8bf288-5965-4eb4-9f9d-bf9b505d4aef None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.413551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8bf288-5965-4eb4-9f9d-bf9b505d4aef None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.414885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8bf288-5965-4eb4-9f9d-bf9b505d4aef None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:17.416552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8bf288-5965-4eb4-9f9d-bf9b505d4aef None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:17.418724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 107/209] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:17 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:17.425030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:17.461404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:17.550786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b22a6-af4b-42fc-bb2c-d23c319d35c9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:17.589867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e55407b-9dc9-413b-b6d8-cfd9852dd327 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.590201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e55407b-9dc9-413b-b6d8-cfd9852dd327 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.590408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e55407b-9dc9-413b-b6d8-cfd9852dd327 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.591898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 108/210] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:17.597814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:17.598730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.599241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.599706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.599873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.600123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.600396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.601282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] RBAC: Adding query filter params (role.id=55face8aaf314343b721a50b5cabf886, scope.project.id=fb4febd431cc435cbade69ba4716e3ec, user.id=cb200a44e3e34c12b3c405c33d703907) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:17.601797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:17.603466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:17.615467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 103/211] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:17 2022] GET /identity/v3/role_assignments?user.id=cb200a44e3e34c12b3c405c33d703907&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=55face8aaf314343b721a50b5cabf886 => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:17.628282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:17.629871 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:17.688229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:17.705600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:17.708874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 109/212] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:17.856681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:17.860778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af692ed-5fd4-438e-aed6-408d229b7dac None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:17.895246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:17.895880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:17.896162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:17.896437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:17.907890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:17.913221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:17.914100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:17.914875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:17.919128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 104/213] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:17.920800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.029705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd12083-188f-4fda-93d1-bbaf3d077deb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.068192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.068804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.069081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.069349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.070017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:18.070318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:18.071708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:18.080191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 110/214] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:18.086071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.097380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.161796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:18.203782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a7ea7-ea27-4a11-b5fe-016ef5cfa2e6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.226387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb80188d-9769-4e5b-97e1-f8a277ea8b36 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.243732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd021c4d-685d-4380-b55f-79d7724a04c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.244376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd021c4d-685d-4380-b55f-79d7724a04c2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.244653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd021c4d-685d-4380-b55f-79d7724a04c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.244925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd021c4d-685d-4380-b55f-79d7724a04c2 None admin] PATH_INFO: `/v3/users/d61e6599f9fa47128171704217641331` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.259619 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd021c4d-685d-4380-b55f-79d7724a04c2 None admin] RBAC: Authorizing `identity:get_user(user_id=d61e6599f9fa47128171704217641331)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:18.261188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd021c4d-685d-4380-b55f-79d7724a04c2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:18.263025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 105/215] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/users/d61e6599f9fa47128171704217641331 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:18.264165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf227d9-1386-4e3d-8935-f2407355dad6 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.264446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf227d9-1386-4e3d-8935-f2407355dad6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.264715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf227d9-1386-4e3d-8935-f2407355dad6 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.266013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 106/216] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:18.269160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf227d9-1386-4e3d-8935-f2407355dad6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.272421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a97454-6df4-402a-aec4-cf948bd6ae2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.273260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a97454-6df4-402a-aec4-cf948bd6ae2f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.273540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a97454-6df4-402a-aec4-cf948bd6ae2f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.273808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a97454-6df4-402a-aec4-cf948bd6ae2f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.274925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a97454-6df4-402a-aec4-cf948bd6ae2f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:18.276400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a97454-6df4-402a-aec4-cf948bd6ae2f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:18.277954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 111/217] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:18.277954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.277954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.277954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.290077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:18.300156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:18.301368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:18.371979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 112/218] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:18.383112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.389504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf227d9-1386-4e3d-8935-f2407355dad6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.433722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.434584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.435375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:18.436626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.437110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.505696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3c8013-5975-48e4-aefd-9f0e9c946bd9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.540137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:18.542350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:18.549956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e016d2a-b0a0-4dbe-b1fe-25e53f29542b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.550731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e016d2a-b0a0-4dbe-b1fe-25e53f29542b None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.551103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e016d2a-b0a0-4dbe-b1fe-25e53f29542b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.551557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e016d2a-b0a0-4dbe-b1fe-25e53f29542b None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.553089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e016d2a-b0a0-4dbe-b1fe-25e53f29542b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:18.555103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e016d2a-b0a0-4dbe-b1fe-25e53f29542b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:18.563297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:18.588084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 113/219] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:18 2022] POST /identity/v3/users => generated 297 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:18.613582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097b3ccd-aa9f-446f-8c5c-5a5066941f27 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.613912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097b3ccd-aa9f-446f-8c5c-5a5066941f27 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.614130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097b3ccd-aa9f-446f-8c5c-5a5066941f27 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.615751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 114/220] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:18.629002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.629822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.630814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.656537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:18.657602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:18.681614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 107/221] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:18.682788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.741231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 115/222] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:18.742470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.782149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:18.805650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbe065c-f51b-431e-8759-69d25dc501a2 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.832540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:18.863803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.865100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.865548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.866096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.869827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10966d0-f1fe-46ab-9292-8b9056bf070a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:18.878437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:18.885286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:18.886167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:18.886983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:18.891226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 108/223] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:18.894108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:18.913111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:18.913782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:18.914068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:18.915452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:18.916432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:18.917877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:18.920650 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 116/224] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:18.927487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.031831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143865ee-970a-4b29-b62d-432f64d3ee63 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.054186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:19.056617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ad6d73-8142-4d37-8c12-5f68338dfd8c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.074478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.075375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.076590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.076874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.077711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:19.078020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.079468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.088570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 109/225] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:19.089417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.104077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.104953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.105234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.105509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.106319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=d61e6599f9fa47128171704217641331) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:19.106633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.108752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.115393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 117/226] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:18 2022] GET /identity/v3/role_assignments?user.id=d61e6599f9fa47128171704217641331&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=807749e5ca90423999feeb8a7727599c => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:19.116767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.165860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:19.249425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a579468c-5113-4484-9abd-332bcc58ba84 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.270723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48d142c-7c79-4b4f-a75c-6a15ab44bc18 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.323219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.323793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.324074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.324357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] PATH_INFO: `/v3/users/b462612c9758449b8a37ab55e52c0570` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.325788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] RBAC: Authorizing `identity:get_user(user_id=b462612c9758449b8a37ab55e52c0570)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.327724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.330068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 118/227] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:19 2022] GET /identity/v3/users/b462612c9758449b8a37ab55e52c0570 => generated 283 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:19.336000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.352495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.353143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.353421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.353693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.367237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:19.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:19.374054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:19.374843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 110/228] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:19.375548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:19.380395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.495118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b0f1c3-716d-4526-a6a9-720801c68782 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.539590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:19.549994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e61457-4fa0-4bd4-921e-dbb73b7768cd None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.554507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.555515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.556042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.556512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.557882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.559903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.562315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 119/229] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:19 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:19.568083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.619337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.620393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.620869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.621335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.622359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:19.624508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.627731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.644368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 111/230] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:19.653580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.780095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:19.784928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc019bd2-9de7-4390-8da9-aae616583d7a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.790109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4651403-4d0b-447d-8e2a-b953142268ee None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:19.850711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.852397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.852682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.852958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/b462612c9758449b8a37ab55e52c0570/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.855550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:19.856177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:19.856448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:19.856720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] PATH_INFO: `/v3/users/b462612c9758449b8a37ab55e52c0570` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:19.859083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] RBAC: Authorizing `identity:create_grant(project_id=d83b0cd39a3c4abebd2479068e9d13c4, user_id=b462612c9758449b8a37ab55e52c0570, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.860529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.863503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] RBAC: Authorizing `identity:get_user(user_id=b462612c9758449b8a37ab55e52c0570)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:19.865011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:19.866688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 112/231] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:19 2022] GET /identity/v3/users/b462612c9758449b8a37ab55e52c0570 => generated 283 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:19.874893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:19.892543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 120/232] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:19 2022] PUT /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/b462612c9758449b8a37ab55e52c0570/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:19.924025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:20.025058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df787628-afdb-4fd5-a7bf-079e26eea31d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:20.074608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba35e8-7acb-43c9-a021-6bd39e2a3745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:20.075297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba35e8-7acb-43c9-a021-6bd39e2a3745 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:20.075579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba35e8-7acb-43c9-a021-6bd39e2a3745 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:20.075922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba35e8-7acb-43c9-a021-6bd39e2a3745 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:20.077188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba35e8-7acb-43c9-a021-6bd39e2a3745 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:20.078725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba35e8-7acb-43c9-a021-6bd39e2a3745 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:20.080869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 113/233] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:19 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:20.086364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:20.205818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757581ca-3868-4d6d-9fda-80fe0b7822ca None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:20.264411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75beacfa-b0d0-49de-82f6-337f2028aa42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:20.265095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75beacfa-b0d0-49de-82f6-337f2028aa42 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:20.265419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75beacfa-b0d0-49de-82f6-337f2028aa42 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:20.265698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75beacfa-b0d0-49de-82f6-337f2028aa42 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/b462612c9758449b8a37ab55e52c0570/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:20.269933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75beacfa-b0d0-49de-82f6-337f2028aa42 None admin] RBAC: Authorizing `identity:create_grant(project_id=d83b0cd39a3c4abebd2479068e9d13c4, user_id=b462612c9758449b8a37ab55e52c0570, role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:20.271522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75beacfa-b0d0-49de-82f6-337f2028aa42 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:20.294066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 121/234] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:20 2022] PUT /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/b462612c9758449b8a37ab55e52c0570/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:25:21.436619 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f2938-798b-41cc-ae6f-6a841f8c888f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:21.437136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f2938-798b-41cc-ae6f-6a841f8c888f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:21.437136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f2938-798b-41cc-ae6f-6a841f8c888f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:21.438763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 114/235] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:21.443764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4e474-9188-4c53-b0f3-02bc81a8082f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:21.444197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4e474-9188-4c53-b0f3-02bc81a8082f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:21.444339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4e474-9188-4c53-b0f3-02bc81a8082f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:21.467924 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-baa4e474-9188-4c53-b0f3-02bc81a8082f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:21.469124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-baa4e474-9188-4c53-b0f3-02bc81a8082f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:21.484870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:21.544418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 122/236] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:21.651605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a762d7-2aad-40ac-9c5d-c8418598ad7f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:21.651895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a762d7-2aad-40ac-9c5d-c8418598ad7f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:21.652173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a762d7-2aad-40ac-9c5d-c8418598ad7f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:21.654285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 115/237] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:21.659618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:21.659906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:21.660179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:21.678694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:21.679894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:21.735588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da58567-579c-480f-ba76-1943a58c71df None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:21.735874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da58567-579c-480f-ba76-1943a58c71df None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:21.736144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da58567-579c-480f-ba76-1943a58c71df None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:21.737627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 116/238] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:21.742647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:21.742948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:21.743233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:21.770542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:21.771790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:21.773958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 123/239] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:21.855176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 117/240] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:21.947146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.037050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:22.057512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12a735d-2ade-4382-9e24-f58804499861 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.092093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.092733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.093046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.093326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.094433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:22.096031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:22.097889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 124/241] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:22.103097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.144593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:22.157472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.266503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0610f6-d828-457b-881c-f1122055ca9d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.315897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.316752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.317432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.317711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.318649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:22.319875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf306ea9-b5e5-4685-99c3-a87f3704f6a2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.320140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:22.350160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 118/242] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:22 2022] POST /identity/v3/users => generated 293 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:22.351090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.369443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.370136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.370479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.370754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.384381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:22.389838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:22.390774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:22.391572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:22.392163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 125/243] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:22.398923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.424776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:22.495204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:22.519847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94098d5a-9bcf-47cc-9974-92bbce0b369f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.557389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c0b56a-6771-4a0d-b9fd-289c5b4c91d7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.566207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccd0e9e-caf7-4064-9431-7c9b227f870b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.566852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccd0e9e-caf7-4064-9431-7c9b227f870b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.567555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccd0e9e-caf7-4064-9431-7c9b227f870b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.567555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccd0e9e-caf7-4064-9431-7c9b227f870b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.579946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccd0e9e-caf7-4064-9431-7c9b227f870b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-0ccd0e9e-caf7-4064-9431-7c9b227f870b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:22.586230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:22.587290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:22.588262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:22.590164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 119/244] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:22.597092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8cbcb1-e90b-4396-9518-73fe3ee9ba25 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.597461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8cbcb1-e90b-4396-9518-73fe3ee9ba25 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.597726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8cbcb1-e90b-4396-9518-73fe3ee9ba25 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.599411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 120/245] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:22.600118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8cbcb1-e90b-4396-9518-73fe3ee9ba25 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.600315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.600920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.601373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.601792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.602639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:22.603775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:22.605346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fa82ef-5ddb-4c4b-8910-163b04411b84 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:22.616422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 126/246] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:22.618244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e4602-2970-484d-97ab-090011c79066 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.619047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e4602-2970-484d-97ab-090011c79066 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.619785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e4602-2970-484d-97ab-090011c79066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.642510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f88e4602-2970-484d-97ab-090011c79066 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:22.643740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f88e4602-2970-484d-97ab-090011c79066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:22.735992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 127/247] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:22.737308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e4602-2970-484d-97ab-090011c79066 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.738141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8cbcb1-e90b-4396-9518-73fe3ee9ba25 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.748214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:22.777955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.778615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.778920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.779224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.779901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:22.780210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:22.782034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:22.798955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 121/248] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:22.801745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.806606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:22.866803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e4602-2970-484d-97ab-090011c79066 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.910972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.911649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.911937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.912216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] PATH_INFO: `/v3/users/d61e6599f9fa47128171704217641331` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:22.913417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] RBAC: Authorizing `identity:get_user(user_id=d61e6599f9fa47128171704217641331)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:22.914839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:22.916699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 128/249] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/users/d61e6599f9fa47128171704217641331 => generated 297 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:22.922756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:22.943938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344fb17-99e4-47ce-8768-7444cbb66ac3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:22.992652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:22.993209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:22.993489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:22.993764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] PATH_INFO: `/v3/users/1f6917e9e8f04bf5bf09afe2d50ad041` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.020422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] RBAC: Authorizing `identity:get_user(user_id=1f6917e9e8f04bf5bf09afe2d50ad041)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:23.021876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:23.023586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 122/250] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/users/1f6917e9e8f04bf5bf09afe2d50ad041 => generated 297 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:23.029883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:23.066744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:23.135491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2575e9-0cb9-4774-8df6-b22e7641a592 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:23.151287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:23.207272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:23.207920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.209083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.209083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.212465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:23.214062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:23.219112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 129/251] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:22 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:23.220019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:23.228138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66498e25-79b7-467b-8ae3-ace2722d2424 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:23.275947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d6a51-992b-4f29-a5f9-b80f5de91202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:23.276943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d6a51-992b-4f29-a5f9-b80f5de91202 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.277731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d6a51-992b-4f29-a5f9-b80f5de91202 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.278110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d6a51-992b-4f29-a5f9-b80f5de91202 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.355132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d6a51-992b-4f29-a5f9-b80f5de91202 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:23.356876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d6a51-992b-4f29-a5f9-b80f5de91202 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:23.364408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc44a01-ef9e-4f0b-9e1d-7c96adaad804 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:23.413640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:23.414289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.414573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.414850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.418295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:23.423505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:23.431519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:23.437171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:23.438036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:23.438802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 130/252] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:23.439491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:23.486554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 123/253] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:23.489074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d47685a-eae9-4574-9bca-5c6ce8025976 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.489569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d47685a-eae9-4574-9bca-5c6ce8025976 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.490037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d47685a-eae9-4574-9bca-5c6ce8025976 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.491576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 124/254] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:23.496038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d47685a-eae9-4574-9bca-5c6ce8025976 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:23.574815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea50d535-bbfb-440e-b3e0-db1c0f6b156c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:23.616750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:23.617759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.618263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.618731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/d61e6599f9fa47128171704217641331/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.620605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d47685a-eae9-4574-9bca-5c6ce8025976 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:23.621103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] RBAC: Authorizing `identity:create_grant(project_id=d83b0cd39a3c4abebd2479068e9d13c4, user_id=d61e6599f9fa47128171704217641331, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:23.623888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:23.640173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 131/255] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:23 2022] PUT /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/d61e6599f9fa47128171704217641331/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:23.642170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:23.702101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:23.703266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.703817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.704287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.705706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:23.706881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:23.709472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42cacbc-9765-42c4-9eec-b526eb1c56bc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:23.730300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 125/256] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:23.734346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.734954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.736471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.768111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:23.769321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:23.770997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:23.793450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:23.836143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b419e19e-44ad-4133-aaaf-1be0892ecc69 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:23.858677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 126/257] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:23.859982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:23.892297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:23.893734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:23.893734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:23.893734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:23.895701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:23.897357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:23.900430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 132/258] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:23 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:23.907525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:24.022329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac25d30c-c436-4ff2-95ee-a34eb1c1dd25 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:24.085755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7e8429-4504-4d6b-a93a-c096ec143ae1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:24.100738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:24.101666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:24.102135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:24.102596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] PATH_INFO: `/v3/users/b462612c9758449b8a37ab55e52c0570` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:24.107666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] RBAC: Authorizing `identity:get_user(user_id=b462612c9758449b8a37ab55e52c0570)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:24.109433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:24.111323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 127/259] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:23 2022] GET /identity/v3/users/b462612c9758449b8a37ab55e52c0570 => generated 283 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:24.116766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:24.123163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:24.143693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:24.144311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:24.144590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:24.144887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:24.145600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] RBAC: Adding query filter params (role.id=55face8aaf314343b721a50b5cabf886, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=1f6917e9e8f04bf5bf09afe2d50ad041) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:24.145910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:24.147391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:24.157500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 133/260] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:23 2022] GET /identity/v3/role_assignments?user.id=1f6917e9e8f04bf5bf09afe2d50ad041&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=55face8aaf314343b721a50b5cabf886 => generated 259 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:24.162120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:24.311700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c9162a-43f8-4bfd-b018-c76a202f6c05 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:24.364343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:24.364876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:24.365151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:24.365481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:24.457001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:24.458834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:24.460438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:24.507292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:24.625596 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b6eeb-4431-4917-a5a4-ec101953106a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:24.662540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 128/261] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:24.670258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:24.677191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:24.677832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:24.678116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:24.678392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:24.697255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:24.713349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:24.714306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:24.715137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 134/262] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:24.727184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:24.817876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:24.866829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d282521-fb71-490c-9f13-102ef47e0907 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:24.894727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:24.917241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:24.918142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:24.918704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:24.920753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eccba9c-702f-4716-9a6a-8be976ca6ee8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:24.921205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:24.922835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:24.925548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:24.932956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 129/263] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:24 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:24.934227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:24.988432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:24.989715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:24.991019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:24.993105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:24.993921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:24.994389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:24.995934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:25.018554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 135/264] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:25.021479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:25.137576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3b7167-2c25-436a-afb7-382ac2acd863 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:25.206982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:25.207837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.208255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.208662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.209542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=b462612c9758449b8a37ab55e52c0570) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:25.210035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:25.211635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ec055-0ec3-4c61-bac2-962e4965280a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:25.221955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8301689-a130-4ce3-ae98-d4f8e3b4131c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:25.225396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 130/265] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:24 2022] GET /identity/v3/role_assignments?user.id=b462612c9758449b8a37ab55e52c0570&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:25.226756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ab69c-9248-4d82-8f84-f7da7bd2d62e None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.227898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ab69c-9248-4d82-8f84-f7da7bd2d62e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.228332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ab69c-9248-4d82-8f84-f7da7bd2d62e None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.230647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 131/266] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:25.237464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.237927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.238350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.252544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:25.285193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:25.287964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:25.319960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:25.320609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.320888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.321159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] PATH_INFO: `/v3/users/b462612c9758449b8a37ab55e52c0570` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.324518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] RBAC: Authorizing `identity:get_user(user_id=b462612c9758449b8a37ab55e52c0570)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:25.326634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:25.338702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 136/267] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:25 2022] GET /identity/v3/users/b462612c9758449b8a37ab55e52c0570 => generated 283 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:25.356459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:25.407529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:25.440203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 132/268] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:25.503042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8a57bd-e2cc-4739-842b-ccb3f40f21e0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:25.541734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb555d3-9764-470a-a8ec-16b68cca49f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:25.542695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb555d3-9764-470a-a8ec-16b68cca49f3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.543242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb555d3-9764-470a-a8ec-16b68cca49f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.543739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb555d3-9764-470a-a8ec-16b68cca49f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.596260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:25.624339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb555d3-9764-470a-a8ec-16b68cca49f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:25.626280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb555d3-9764-470a-a8ec-16b68cca49f3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:25.738283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:25.750685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 137/269] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:25.753397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149d1510-45a9-4faf-bb75-2b1e3369be00 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.753617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149d1510-45a9-4faf-bb75-2b1e3369be00 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.753884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149d1510-45a9-4faf-bb75-2b1e3369be00 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.755349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 138/270] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:25.756056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149d1510-45a9-4faf-bb75-2b1e3369be00 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:25.857088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:25.890319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7607fc1e-dc4a-4d81-b11a-85cbf8c23530 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:25.890546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149d1510-45a9-4faf-bb75-2b1e3369be00 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:25.930359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ba25bb-c062-418f-9d2a-1a0d8f0c99a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:25.931025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ba25bb-c062-418f-9d2a-1a0d8f0c99a6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.931308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ba25bb-c062-418f-9d2a-1a0d8f0c99a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.931583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ba25bb-c062-418f-9d2a-1a0d8f0c99a6 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.932195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:25.932708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ba25bb-c062-418f-9d2a-1a0d8f0c99a6 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:25.933622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.934099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.934425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ba25bb-c062-418f-9d2a-1a0d8f0c99a6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:25.934843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.936174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:25.936589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 139/271] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:25 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:25.937767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:25.938059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:25.938333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:25.938798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:25.940659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 133/272] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:25.942369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:25.959498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:25.961453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:26.012049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:26.043605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 140/273] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:26.044668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:26.063692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2892fac9-d63e-4615-ab5f-56a81b4fbb3d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:26.107645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:26.108263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.108544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.109354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.110226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] RBAC: Adding query filter params (role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=b462612c9758449b8a37ab55e52c0570) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:26.110688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:26.112830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:26.121188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 134/274] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:25 2022] GET /identity/v3/role_assignments?user.id=b462612c9758449b8a37ab55e52c0570&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:26.158036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:26.197290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fc82c4-4db8-4846-995b-d144cc581cf1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:26.255384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:26.256009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.256288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.256562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.257234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:26.258704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:26.293701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 141/275] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:26 2022] POST /identity/v3/users => generated 293 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:26.398542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:26.517986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5aff5a-a028-4964-a455-e1fcff9a6902 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:26.537335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:26.561482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:26.562024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.562307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.562580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.574000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:26.579584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:26.580531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:26.581342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 135/276] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:26.586821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:26.711441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec10789-75a6-4d64-b3d4-b65d66d82567 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:26.743416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:26.750237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:26.751403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.751745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.752023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.752744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:26.753050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:26.754477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d836ed51-93cf-48ac-9cd6-91b46187c5fc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:26.764991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 142/277] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:26.770820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:26.801433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdcb1a7-c2ed-4b0b-8f43-1fb636960104 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.801771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdcb1a7-c2ed-4b0b-8f43-1fb636960104 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.802058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdcb1a7-c2ed-4b0b-8f43-1fb636960104 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.803729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 143/278] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:26.809497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.810299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.810584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.833345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:26.834789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:26.897885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840cf78c-f604-4a5d-8ff6-19b654279fcc None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:26.929571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 144/279] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:26.944773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:26.945416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:26.945701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:26.945980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] PATH_INFO: `/v3/users/02da2936d99d46d39e79875a6c0dfecf` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:26.960728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] RBAC: Authorizing `identity:get_user(user_id=02da2936d99d46d39e79875a6c0dfecf)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:26.962226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:26.964203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 136/280] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:26 2022] GET /identity/v3/users/02da2936d99d46d39e79875a6c0dfecf => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:26.970412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:27.110078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bedbe-e9c4-41bd-893b-4a98bfa34ae5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:27.143911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:27.162224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:27.162851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:27.163164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:27.163442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:27.164184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, user.id=02da2936d99d46d39e79875a6c0dfecf, scope.system=all) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:27.164498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:27.166123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccd10d-8276-4cf5-af48-135996e92ca1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:27.172438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 145/281] 10.222.0.20 () {56 vars in 1308 bytes} [Fri Sep 9 04:25:26 2022] GET /identity/v3/role_assignments?user.id=02da2936d99d46d39e79875a6c0dfecf&scope.system=all&role.id=807749e5ca90423999feeb8a7727599c => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:27.248338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:27.286080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:27.351939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed6c4c-7b59-4d16-bbba-bba3dc3ce955 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:27.352225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed6c4c-7b59-4d16-bbba-bba3dc3ce955 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:27.352501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed6c4c-7b59-4d16-bbba-bba3dc3ce955 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:27.353998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 146/282] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:27.371977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:27.372246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:27.372523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:27.385668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82142f30-0ddc-48f2-b8be-aeafc437d12d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:27.413792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:27.416808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:27.454870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:27.455685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:27.455922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:27.456198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:27.470002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:27.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:27.475917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:27.476698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 137/283] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:27.489370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:27.532878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 147/284] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:27.627467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:27.636166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d27ae-6d74-4497-a95a-afe59e55bf89 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:27.686821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:27.687847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:27.688331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:27.688823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:27.689684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:27.690182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:27.691779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:27.706357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 138/285] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:27.712288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:27.851775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca45def-3bb0-4f2e-82e7-1c94b2c9849a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:27.861125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:27.876392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:27.896932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:27.898060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:27.898603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:27.899103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] PATH_INFO: `/v3/users/d61e6599f9fa47128171704217641331` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:27.900455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] RBAC: Authorizing `identity:get_user(user_id=d61e6599f9fa47128171704217641331)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:27.902416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:27.904763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 148/286] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:27 2022] GET /identity/v3/users/d61e6599f9fa47128171704217641331 => generated 297 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:27.910981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.018859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b9b9f0-fadd-4609-becd-71dc16cacc7e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.056681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:28.057696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:28.059171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:28.059775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:28.102483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2142721-6a0c-44a5-9dd1-9229661143d4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.168757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:28.169964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:28.172707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:28.174056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:28.174765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:28.194908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:28.201716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:28.203508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:28.205117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:28.206161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:28.206237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 139/287] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 343 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:28.206303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:28.213139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.251305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:28.402606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 149/288] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:28.412299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2080b0-268f-4f3e-9d65-f0fa801782e2 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.415231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.471701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:28.472659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:28.473139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:28.473611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:28.474480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] RBAC: Adding query filter params (name=reader) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:28.475023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:28.476607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:28.485700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 140/289] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:28.491375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.573188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:28.599694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:28.601569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561a0ca-4e2a-4de6-bd37-52fd1a88ba21 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.647186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614fb838-30f5-4bee-9167-d9d03cccd9bc None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.678047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:28.679266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:28.680606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:28.682480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:28.686750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:28.691571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:28.693608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 150/290] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:28 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:28.699714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.722749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:28.723721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:28.724147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:28.724618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] PATH_INFO: `/v3/users/1f6917e9e8f04bf5bf09afe2d50ad041` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:28.725947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] RBAC: Authorizing `identity:get_user(user_id=1f6917e9e8f04bf5bf09afe2d50ad041)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:28.728068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:28.730660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 141/291] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:28 2022] GET /identity/v3/users/1f6917e9e8f04bf5bf09afe2d50ad041 => generated 297 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:28.737357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.904167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe10ee-8d23-4dbd-ae27-f5b206c3a143 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.941276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189cf30c-61c0-42fc-a06f-fdb34c369303 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:28.944202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:28.945152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:28.945584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:28.946054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:28.947506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:28.950959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:28.953068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 142/292] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:28 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:28.960330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:28.980759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:29.037032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:29.037656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.037936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.038206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.038964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=d61e6599f9fa47128171704217641331) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:29.039280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:29.040728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8cb3de-d878-4f60-81dc-7e1df8ec0ee8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:29.056051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 151/293] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:28 2022] GET /identity/v3/role_assignments?user.id=d61e6599f9fa47128171704217641331&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=807749e5ca90423999feeb8a7727599c => generated 608 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:29.066469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bac7b6-fd2b-415c-8304-175c3939c652 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.066754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bac7b6-fd2b-415c-8304-175c3939c652 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.067057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bac7b6-fd2b-415c-8304-175c3939c652 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.068663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 152/294] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:29.084634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b50bca-c196-45c1-b22e-57780ff967c1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:29.084861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d14e1c5-33a2-4d87-98d9-b8831188d4c8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.085060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d14e1c5-33a2-4d87-98d9-b8831188d4c8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.085341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d14e1c5-33a2-4d87-98d9-b8831188d4c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.124036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5d14e1c5-33a2-4d87-98d9-b8831188d4c8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:29.125253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5d14e1c5-33a2-4d87-98d9-b8831188d4c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:29.128603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1415e2-f900-414a-9b7b-2686b30c04d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:29.129552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1415e2-f900-414a-9b7b-2686b30c04d5 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.130027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1415e2-f900-414a-9b7b-2686b30c04d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.130442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1415e2-f900-414a-9b7b-2686b30c04d5 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/1f6917e9e8f04bf5bf09afe2d50ad041/roles/55face8aaf314343b721a50b5cabf886` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.133927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1415e2-f900-414a-9b7b-2686b30c04d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=d83b0cd39a3c4abebd2479068e9d13c4, user_id=1f6917e9e8f04bf5bf09afe2d50ad041, role_id=55face8aaf314343b721a50b5cabf886)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:29.135970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1415e2-f900-414a-9b7b-2686b30c04d5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:29.149111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:29.163717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 143/295] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:25:28 2022] PUT /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/users/1f6917e9e8f04bf5bf09afe2d50ad041/roles/55face8aaf314343b721a50b5cabf886 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:25:29.250023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 153/296] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:29.365811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e402eeb-376e-405c-8421-254cea395056 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.366336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e402eeb-376e-405c-8421-254cea395056 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.366761 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e402eeb-376e-405c-8421-254cea395056 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.368378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 144/297] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:29.375704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc094e3f-8506-452a-8eb9-90286232962a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.375988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc094e3f-8506-452a-8eb9-90286232962a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.376263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc094e3f-8506-452a-8eb9-90286232962a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.397251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bc094e3f-8506-452a-8eb9-90286232962a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:29.398454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bc094e3f-8506-452a-8eb9-90286232962a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:29.484464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 154/298] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:29.587034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e402eeb-376e-405c-8421-254cea395056 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:29.632505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080d0c0-67fe-4052-9e88-3b98aeded855 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.632505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080d0c0-67fe-4052-9e88-3b98aeded855 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.632752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080d0c0-67fe-4052-9e88-3b98aeded855 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.634229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 155/299] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:29.639642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.639928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.640204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.660723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:29.662179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:29.675511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:29.704344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e402eeb-376e-405c-8421-254cea395056 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:29.743435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:29.744082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.744342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.744616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.752369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 156/300] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:29.758079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:29.763678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:29.764552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:29.765329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 145/301] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:29.772117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:29.773458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:29.783870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:29.885135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb123c-b49c-4120-8745-03f152fbd7ab None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:29.888414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fbdc96-0ced-40a9-8713-eb01f469d7b3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:29.921624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:29.922285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.922515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.922785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.924296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:29.925753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:29.927533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 146/302] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:29.929435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:29.930255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:29.930725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:29.931224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:29.932170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:29.932650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:29.932875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:29.935354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:29.936127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:29.947166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 157/303] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:29.956307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:30.029211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:30.042427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb718e0-f96a-4b18-99d6-bddb44539a57 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.064301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf335a2-c63e-4318-aac4-40cc463156f0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.078270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.078877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.079187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.079457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] PATH_INFO: `/v3/groups` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.080087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:30.081526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:30.099584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 147/304] 10.222.0.20 () {60 vars in 1134 bytes} [Fri Sep 9 04:25:29 2022] POST /identity/v3/groups => generated 229 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:25:30.100236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:30.104323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.105323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.105824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.106098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] PATH_INFO: `/v3/users/dd9fb67355d4492ea91947688697a903` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.122736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] RBAC: Authorizing `identity:get_user(user_id=dd9fb67355d4492ea91947688697a903)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:30.124330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:30.131138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 158/305] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:29 2022] GET /identity/v3/users/dd9fb67355d4492ea91947688697a903 => generated 293 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:30.132025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:30.197884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:30.249920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6200600-689b-4ffb-ab50-4d3367a9e585 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.268177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c2180c-4c5a-4d19-a00a-0a4254abc2a6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.283779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:30.301280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.301920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.302209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.302495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.303865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] RBAC: Adding query filter params (role.id=55face8aaf314343b721a50b5cabf886, user.id=dd9fb67355d4492ea91947688697a903, scope.system=all) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:30.303865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:30.306335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:30.314070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 159/306] 10.222.0.20 () {56 vars in 1308 bytes} [Fri Sep 9 04:25:30 2022] GET /identity/v3/role_assignments?user.id=dd9fb67355d4492ea91947688697a903&scope.system=all&role.id=55face8aaf314343b721a50b5cabf886 => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:30.321652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.322317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.322595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.322864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.336048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:30.341891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:30.342864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:30.343683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 148/307] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:30.350798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:30.463516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1afe48b-3f35-48b6-a36e-e0848ee125d9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.501253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.501865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.502139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.502405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.503106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:30.503414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:30.504782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:30.514476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 160/308] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:30.520080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:30.631126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7a45cb-3aea-4388-a1c3-93cd820b7aae None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.655363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:30.667454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e019278-e5df-4406-a47e-7efae4545d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.668090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e019278-e5df-4406-a47e-7efae4545d1f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.668373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e019278-e5df-4406-a47e-7efae4545d1f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.668649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e019278-e5df-4406-a47e-7efae4545d1f None admin] PATH_INFO: `/v3/users/02da2936d99d46d39e79875a6c0dfecf` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.669869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e019278-e5df-4406-a47e-7efae4545d1f None admin] RBAC: Authorizing `identity:get_user(user_id=02da2936d99d46d39e79875a6c0dfecf)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:30.671358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e019278-e5df-4406-a47e-7efae4545d1f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:30.673234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 149/309] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:30 2022] GET /identity/v3/users/02da2936d99d46d39e79875a6c0dfecf => generated 293 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:30.678571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:30.739945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:30.786134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868243e-8328-4777-bdc2-105a08a3cdcb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:30.822434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb1efc0-928b-4b2d-a751-2a105208ab6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:30.823110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb1efc0-928b-4b2d-a751-2a105208ab6a None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:30.823391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb1efc0-928b-4b2d-a751-2a105208ab6a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:30.823666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb1efc0-928b-4b2d-a751-2a105208ab6a None admin] PATH_INFO: `/v3/system/users/02da2936d99d46d39e79875a6c0dfecf/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:30.825111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb1efc0-928b-4b2d-a751-2a105208ab6a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=02da2936d99d46d39e79875a6c0dfecf, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:30.826558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb1efc0-928b-4b2d-a751-2a105208ab6a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:30.837246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 161/310] 10.222.0.20 () {58 vars in 1334 bytes} [Fri Sep 9 04:25:30 2022] PUT /identity/v3/system/users/02da2936d99d46d39e79875a6c0dfecf/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:31.501944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed7432-8b8b-4a8f-99e0-31ae6a71f78e None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:31.503040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed7432-8b8b-4a8f-99e0-31ae6a71f78e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:31.503556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed7432-8b8b-4a8f-99e0-31ae6a71f78e None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:31.505363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 150/311] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:31.510412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166909b-dbb0-427b-b326-8e785366162a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:31.510869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166909b-dbb0-427b-b326-8e785366162a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:31.510974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166909b-dbb0-427b-b326-8e785366162a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:31.530670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b166909b-dbb0-427b-b326-8e785366162a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:31.532073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b166909b-dbb0-427b-b326-8e785366162a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:31.603785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 162/312] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:31.874924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ed7432-8b8b-4a8f-99e0-31ae6a71f78e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:31.985293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ed7432-8b8b-4a8f-99e0-31ae6a71f78e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:32.020578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:32.021593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.022072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.022560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.033969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:32.039206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:32.040086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:32.040883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:32.043106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 151/313] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:32.050834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166909b-dbb0-427b-b326-8e785366162a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:32.099104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:32.158171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166909b-dbb0-427b-b326-8e785366162a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:32.201269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:32.202346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.202720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.203026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.203716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:32.204024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:32.206089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:32.216342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 163/314] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:32.221624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:32.273853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:32.327123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d2755-8e4d-4b36-8a52-9f8de80da61b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:32.362140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23098ddd-f281-4295-a74a-9f848adae423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:32.362786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23098ddd-f281-4295-a74a-9f848adae423 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.363102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23098ddd-f281-4295-a74a-9f848adae423 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.363377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23098ddd-f281-4295-a74a-9f848adae423 None admin] PATH_INFO: `/v3/users/1f6917e9e8f04bf5bf09afe2d50ad041` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.364584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23098ddd-f281-4295-a74a-9f848adae423 None admin] RBAC: Authorizing `identity:get_user(user_id=1f6917e9e8f04bf5bf09afe2d50ad041)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:32.366360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23098ddd-f281-4295-a74a-9f848adae423 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:32.368538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 152/315] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:32 2022] GET /identity/v3/users/1f6917e9e8f04bf5bf09afe2d50ad041 => generated 297 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:32.375063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:32.491044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0370fc9-7ad2-4b88-8db0-531e3f6ef021 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:32.491217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6cc1e5-2d4b-475b-b51c-5248d94dc46e None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.491217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6cc1e5-2d4b-475b-b51c-5248d94dc46e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.491217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6cc1e5-2d4b-475b-b51c-5248d94dc46e None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.492597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 153/316] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:32.496584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735b15b2-72cc-4438-98e9-6554cc841b1c None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.496942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735b15b2-72cc-4438-98e9-6554cc841b1c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.497224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735b15b2-72cc-4438-98e9-6554cc841b1c None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.498598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 154/317] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:32.499992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:32.501340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0246c9-a454-417d-a93e-9d8ca470ae3d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.501340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0246c9-a454-417d-a93e-9d8ca470ae3d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.501593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0246c9-a454-417d-a93e-9d8ca470ae3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.521274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-1b0246c9-a454-417d-a93e-9d8ca470ae3d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:32.522480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1b0246c9-a454-417d-a93e-9d8ca470ae3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:32.525224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:32.525839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.526123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.526401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.593762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 155/318] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:32.595466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.595771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.596044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.611264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:32.613151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:32.613429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:32.614601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:32.661132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:32.695355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 156/319] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:32.739091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 164/320] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:32.744408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:32.855607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971b87a4-ccf6-4254-8ffa-79be931ff78f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:32.866478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:32.880591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:32.897644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:32.898541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:32.898831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:32.899596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:32.900977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:32.903683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:32.906157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 157/321] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:32 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:32.911402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:32.972739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:32.982325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d776e4b-200f-4388-a808-f33b23e3e70b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.021117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.021691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.021969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.022243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.023488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:33.024986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:33.026673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 165/322] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:33.027506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.035995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b6e10-a929-44cd-b326-5f5a7e33ee84 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.075179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.075796 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.076076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.076348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.077078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] RBAC: Adding query filter params (role.id=55face8aaf314343b721a50b5cabf886, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4, user.id=1f6917e9e8f04bf5bf09afe2d50ad041) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:33.077388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:33.078869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:33.094959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 158/323] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:25:32 2022] GET /identity/v3/role_assignments?user.id=1f6917e9e8f04bf5bf09afe2d50ad041&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=55face8aaf314343b721a50b5cabf886 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:33.094959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.136095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:33.166744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725d0ca8-303a-44aa-a659-dd73653ae77e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.207305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04649a-7eeb-4761-a6dd-4c1d6d9f3253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.208221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04649a-7eeb-4761-a6dd-4c1d6d9f3253 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.208507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04649a-7eeb-4761-a6dd-4c1d6d9f3253 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.208780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04649a-7eeb-4761-a6dd-4c1d6d9f3253 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.216298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9729de-4ac9-43c0-a99f-09f39f0aaaf5 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.221168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d04649a-7eeb-4761-a6dd-4c1d6d9f3253 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6d04649a-7eeb-4761-a6dd-4c1d6d9f3253 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:33.226256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:33.227204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:33.227993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 166/324] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:33.229759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4f80d-9d4d-4eb1-9746-8c1ab68f2ab3 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.229759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4f80d-9d4d-4eb1-9746-8c1ab68f2ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.229952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4f80d-9d4d-4eb1-9746-8c1ab68f2ab3 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.231695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 167/325] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:33.233608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:33.238512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4f80d-9d4d-4eb1-9746-8c1ab68f2ab3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.266474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b120d167-4245-4937-bdec-256c9e94baab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.267131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b120d167-4245-4937-bdec-256c9e94baab None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.267417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b120d167-4245-4937-bdec-256c9e94baab None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.267693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b120d167-4245-4937-bdec-256c9e94baab None admin] PATH_INFO: `/v3/groups` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.268569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b120d167-4245-4937-bdec-256c9e94baab None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:33.269959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b120d167-4245-4937-bdec-256c9e94baab None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:33.295073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 159/326] 10.222.0.20 () {60 vars in 1134 bytes} [Fri Sep 9 04:25:33 2022] POST /identity/v3/groups => generated 226 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:33.296322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.296609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.296882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.316435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:33.317666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:33.385831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4f80d-9d4d-4eb1-9746-8c1ab68f2ab3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.418294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 160/327] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:33.432335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.433307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.433584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.433861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.434625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:33.435026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:33.436516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:33.443234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:33.448024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 168/328] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:33.453978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.561181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:33.564006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42717677-7f85-471a-a2a5-0a55733901d7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.604342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.604975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.605255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.605524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] PATH_INFO: `/v3/users/dd9fb67355d4492ea91947688697a903` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.606713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] RBAC: Authorizing `identity:get_user(user_id=dd9fb67355d4492ea91947688697a903)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:33.608197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:33.610074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 161/329] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:33 2022] GET /identity/v3/users/dd9fb67355d4492ea91947688697a903 => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:33.615611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.693867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.728300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941979b6-e7b3-4add-b9e2-bb2f6fa04ab4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.775113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.775783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.776352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.776638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] PATH_INFO: `/v3/system/users/dd9fb67355d4492ea91947688697a903/roles/55face8aaf314343b721a50b5cabf886` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.779195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=dd9fb67355d4492ea91947688697a903, role_id=55face8aaf314343b721a50b5cabf886)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:33.780888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:33.793401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 169/330] 10.222.0.20 () {58 vars in 1334 bytes} [Fri Sep 9 04:25:33 2022] PUT /identity/v3/system/users/dd9fb67355d4492ea91947688697a903/roles/55face8aaf314343b721a50b5cabf886 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:33.826662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f1456-0b6e-425c-8c82-2d07c18c4709 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:33.866092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:33.866817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:33.867135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:33.867410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:33.881107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:33.886420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:33.887493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:33.888325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 162/331] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:33.894101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:33.977907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:34.031711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986816a0-ed64-43dd-afa2-62fb96e9cf3a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:34.067435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:34.068049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:34.068334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:34.068613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:34.069289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:34.069598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:34.071086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:34.081516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 170/332] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:34.087307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:34.126474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:34.216569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdf3c9-dca2-43a2-931d-93051bdc841f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:34.266272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9731d9-dba9-4214-9e3b-ff6cc2c75fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:34.267124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9731d9-dba9-4214-9e3b-ff6cc2c75fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:34.267571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9731d9-dba9-4214-9e3b-ff6cc2c75fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:34.267984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9731d9-dba9-4214-9e3b-ff6cc2c75fc3 None admin] PATH_INFO: `/v3/users/02da2936d99d46d39e79875a6c0dfecf` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:34.271823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9731d9-dba9-4214-9e3b-ff6cc2c75fc3 None admin] RBAC: Authorizing `identity:get_user(user_id=02da2936d99d46d39e79875a6c0dfecf)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:34.273502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9731d9-dba9-4214-9e3b-ff6cc2c75fc3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:34.275642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 163/333] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:34 2022] GET /identity/v3/users/02da2936d99d46d39e79875a6c0dfecf => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:34.281174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:34.392180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb66b380-9d33-483f-8ecd-acd54f615024 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:34.420897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:34.431214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:34.431829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:34.432108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:34.432393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:34.433456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] RBAC: Adding query filter params (role.id=807749e5ca90423999feeb8a7727599c, user.id=02da2936d99d46d39e79875a6c0dfecf, scope.system=all) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:34.433791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:34.435873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6298d-cecc-48fd-8367-90495ec8174c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:34.449021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 171/334] 10.222.0.20 () {56 vars in 1308 bytes} [Fri Sep 9 04:25:34 2022] GET /identity/v3/role_assignments?user.id=02da2936d99d46d39e79875a6c0dfecf&scope.system=all&role.id=807749e5ca90423999feeb8a7727599c => generated 510 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:35.822099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d4cce-2063-495b-915b-37c2c1f4b048 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:35.823179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d4cce-2063-495b-915b-37c2c1f4b048 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:35.823686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d4cce-2063-495b-915b-37c2c1f4b048 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:35.825576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 164/335] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:35.830762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8facbd2b-1b95-42e9-815d-b896a5ae23f2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:35.831695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8facbd2b-1b95-42e9-815d-b896a5ae23f2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:35.832231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8facbd2b-1b95-42e9-815d-b896a5ae23f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:35.859968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8facbd2b-1b95-42e9-815d-b896a5ae23f2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:35.861575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8facbd2b-1b95-42e9-815d-b896a5ae23f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:35.948109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 172/336] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:36.487750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d4cce-2063-495b-915b-37c2c1f4b048 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:36.603240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3638d-55ca-4679-9864-9de4405c3746 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:36.603909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3638d-55ca-4679-9864-9de4405c3746 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:36.604390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3638d-55ca-4679-9864-9de4405c3746 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:36.606508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 173/337] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:36.612792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c86c8af-16c2-4cf8-936e-1411c936273e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:36.613388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c86c8af-16c2-4cf8-936e-1411c936273e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:36.613735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:36.613946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c86c8af-16c2-4cf8-936e-1411c936273e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:36.626537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d4cce-2063-495b-915b-37c2c1f4b048 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:36.647963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5c86c8af-16c2-4cf8-936e-1411c936273e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:36.649399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5c86c8af-16c2-4cf8-936e-1411c936273e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:36.707679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:36.708457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:36.708740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:36.709073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:36.730109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:36.735543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:36.737165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:36.738198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 165/338] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:36.742444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:36.771909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 174/339] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:36.773575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c81252-f6c2-4004-95cb-c83517103ee6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:36.774057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c81252-f6c2-4004-95cb-c83517103ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:36.774568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c81252-f6c2-4004-95cb-c83517103ee6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:36.776022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 175/340] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:36.782809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:36.783346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:36.783818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:36.811378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:36.813007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:36.857263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450bb1d2-f3ba-47f0-8808-f0de8e1c6374 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:36.878566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:36.936581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:36.937207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:36.937486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:36.937760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:36.938435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:36.938739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:36.940163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:36.943462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 176/341] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:36.951665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 166/342] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:36.960111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.062051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5e97c2-e3f6-4bac-a968-c98e19a5f940 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:37.097221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:37.097829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.098108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.098381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.155302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:37.168037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:37.170034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:37.287914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 177/343] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:37.294169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.319343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:37.344605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.438290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:37.479637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a7341-ef41-4058-ae4d-17a714e99685 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:37.508980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7241f3-ca25-49fb-a02d-bd631fb7ef75 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:37.523877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:37.525213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.525498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.525772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.526998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:37.529011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:37.537351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 167/344] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:37 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:37.538360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.640187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:37.640838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.641027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:37.641133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.641380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.654807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31e6e3f-e33a-456f-a12f-b037385c969c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:37.669972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:37.677237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:37.678068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:37.678873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:37.680148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 178/345] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:37.684237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.701212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f02b83-83d0-4916-babd-05b1d67b70c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:37.701776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f02b83-83d0-4916-babd-05b1d67b70c9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.702054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f02b83-83d0-4916-babd-05b1d67b70c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.702325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f02b83-83d0-4916-babd-05b1d67b70c9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.715459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f02b83-83d0-4916-babd-05b1d67b70c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-07f02b83-83d0-4916-babd-05b1d67b70c9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:37.720877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:37.721872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:37.722674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 168/346] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:37.724163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2581e1-5abb-4ed5-b6fe-13ac4e3c91d3 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.724452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2581e1-5abb-4ed5-b6fe-13ac4e3c91d3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.724726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2581e1-5abb-4ed5-b6fe-13ac4e3c91d3 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.726240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 169/347] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:37.729788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2581e1-5abb-4ed5-b6fe-13ac4e3c91d3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.826128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2086739f-c56e-4007-bd43-3f0d9495da00 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:37.853501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2581e1-5abb-4ed5-b6fe-13ac4e3c91d3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:37.876194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:37.876649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.877043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.877392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.880029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:37.885317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:37.887076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:37.894774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 179/348] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:37 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:37.896122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:37.903154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:37.904321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.904791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.905235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.906073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:37.906562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:37.908686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ca635-37b0-43ed-9832-281908d5af8c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:37.920959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 170/349] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:37.922790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:37.923272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:37.924308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:37.949231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:37.951735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:37.955398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.028848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187f06a-fc96-48d2-b5da-dbb0debfd9bd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.037742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 171/350] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:38.039032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.065320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.066026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.066311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.066587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.067294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:38.067601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.068965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.077763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 180/351] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:38.079760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.140896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.152462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec49c81a-57f2-44ba-b159-0a247a590e7c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.187022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb41206f-0b91-4dbd-bc7e-0b5cc653fb67 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.195206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.195892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.196277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.196560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.197352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=fb4febd431cc435cbade69ba4716e3ec) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:38.197766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.199312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7c23c-fe29-40b3-bad7-4dbdf543c761 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.205626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 172/352] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=807749e5ca90423999feeb8a7727599c => generated 260 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:38.206807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779045f5-0f01-4dcc-aa43-a1561b7217a2 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.207116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779045f5-0f01-4dcc-aa43-a1561b7217a2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.207394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779045f5-0f01-4dcc-aa43-a1561b7217a2 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.208711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 173/353] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:38.209429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779045f5-0f01-4dcc-aa43-a1561b7217a2 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.217081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.246040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.247222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.247509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.247781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.380014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779045f5-0f01-4dcc-aa43-a1561b7217a2 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.397263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.398832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.446181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1043c2a-70dd-489e-b297-b1da4aab1a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.446840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1043c2a-70dd-489e-b297-b1da4aab1a75 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.447144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1043c2a-70dd-489e-b297-b1da4aab1a75 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.447475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1043c2a-70dd-489e-b297-b1da4aab1a75 None admin] PATH_INFO: `/v3/users/dd9fb67355d4492ea91947688697a903` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.449166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1043c2a-70dd-489e-b297-b1da4aab1a75 None admin] RBAC: Authorizing `identity:get_user(user_id=dd9fb67355d4492ea91947688697a903)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.451422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1043c2a-70dd-489e-b297-b1da4aab1a75 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.453401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 174/354] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/users/dd9fb67355d4492ea91947688697a903 => generated 293 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:38.454587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.454916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.455218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.481729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:38.483043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:38.491182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.545943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 181/355] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:38.546887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.556523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.563737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 175/356] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:38.563737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.666178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a892-aee3-4a5f-bab3-b5037d7560c1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.686702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fc43a4-be93-4c0a-a4b5-7a15fe8ecdbf None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.702229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bba49b2-aac3-4c30-ace6-190287ddd4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.702861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bba49b2-aac3-4c30-ace6-190287ddd4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.703170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bba49b2-aac3-4c30-ace6-190287ddd4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.703439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bba49b2-aac3-4c30-ace6-190287ddd4b6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.714473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bba49b2-aac3-4c30-ace6-190287ddd4b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8bba49b2-aac3-4c30-ace6-190287ddd4b6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:38.721472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:38.722627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:38.723521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 182/357] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:38.724497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91204826-31a1-45de-85be-ab704def2d89 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.724497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91204826-31a1-45de-85be-ab704def2d89 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.726982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91204826-31a1-45de-85be-ab704def2d89 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.726982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 183/358] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:38.727604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91204826-31a1-45de-85be-ab704def2d89 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.732224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.732781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.733058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.733326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.734180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] RBAC: Adding query filter params (role.id=55face8aaf314343b721a50b5cabf886, user.id=dd9fb67355d4492ea91947688697a903, scope.system=all) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:38.734594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.736083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.751551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 176/359] 10.222.0.20 () {56 vars in 1308 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/role_assignments?user.id=dd9fb67355d4492ea91947688697a903&scope.system=all&role.id=55face8aaf314343b721a50b5cabf886 => generated 510 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:38.752393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:38.761245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.819382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:38.867110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351cfb00-da35-4963-bfbd-8993547a30e9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.903865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91204826-31a1-45de-85be-ab704def2d89 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:38.904398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.904874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.905100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.905340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.906035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:38.906338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.907805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08c7ca-9185-4fed-95b0-915949d1bc1b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.921672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 177/360] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:38.923484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.923982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.924524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.945310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:38.947014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:38.956338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:38.956952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:38.957229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:38.957504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:38.958685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:38.960376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:38.962482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 184/361] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:38.963250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.047751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 178/362] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:39.048811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.098110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.113801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f9f21e-bdd4-4862-a51e-e92ce7e0e4c2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.129329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.174604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.175281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.175565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.175843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.194074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaedf42-4d0f-4083-af29-48d67a832ec7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.195574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:39.201047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:39.201930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:39.202718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 185/363] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:39.207498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.244931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.245649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.245937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.246209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.334681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdd558-8114-4943-bc17-e63cff7cc350 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.341737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:39.343293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:39.386842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.388046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.389229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.389229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.390260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:39.391737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:39.393664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 186/364] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:39.394897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.416692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.421886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.481992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 179/365] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:39.483185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.522255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e23639f-318c-4b4f-9d20-2ec0d5bcf912 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.572713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.573334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.573615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.574193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.575555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:39.575862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:39.577309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:39.586243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 187/366] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:39.587054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.603500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754dcf94-9a7f-45ee-94fe-7178f32068af None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.641617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.642254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.642534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.642807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.657766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:39.662856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:39.663786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:39.664615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 180/367] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:39.665291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.682305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.687769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.715109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551892db-5b61-4547-a202-148ead87d5eb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.765361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.765980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.767518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.767518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.767518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=fb4febd431cc435cbade69ba4716e3ec) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:39.767775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:39.768967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:39.774821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 188/368] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:39.776118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.800193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e72b97-a38b-4e04-8166-c8c6c60fcf7b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.855054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.855746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.856328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.856603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.858127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:39.860473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:39.862589 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 181/369] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:39.863287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:39.923129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2246d70-30ac-45b9-b110-acbcd2944443 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:39.969104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:39.970099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:39.970448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:39.970803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:39.982484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:39.991324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.016410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1700ce5-f221-4bf1-867b-70432e0df3e6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.060054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.060733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.061001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.061325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.061998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:40.062298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.063701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.066580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.068129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.078366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 182/370] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:40.079298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.191186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 189/371] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:40.196520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.213362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9609543e-2845-4f85-94ec-cdc87ca2dc40 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.259256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.259746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.260029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.260305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.261483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.263609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.270413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 183/372] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:40.270413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.271740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.281540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.341409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220b5248-0410-414c-ad0f-c4e406c8535f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.380003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.380618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.380897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.381167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.397673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c0aaa6-eb5e-4b35-b73b-055a571aa052 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.452207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.453437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.454282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.455047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.456609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.459072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.461214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 184/373] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:40.462135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.469976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.474322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.524747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.571923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.603841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d651494-0159-4bee-83fe-28aa84107517 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.606207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 190/374] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:40.609704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.655720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.656334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.656612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.656882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.657626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:40.657937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.661728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.678083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 185/375] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=807749e5ca90423999feeb8a7727599c => generated 260 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:40.679778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.772019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fd83-834d-4274-97d6-68130c38da32 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.820887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844a94c4-2728-4f10-8a85-86e32b375072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.821538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844a94c4-2728-4f10-8a85-86e32b375072 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.821660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d809d107-f89a-42a4-9262-f8edc96fa114 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.821853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844a94c4-2728-4f10-8a85-86e32b375072 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.822086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844a94c4-2728-4f10-8a85-86e32b375072 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.823459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844a94c4-2728-4f10-8a85-86e32b375072 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.826380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844a94c4-2728-4f10-8a85-86e32b375072 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.828394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 191/376] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:40.829619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8ad9-8a57-4ac9-9c5f-99c242719ec5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.829905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8ad9-8a57-4ac9-9c5f-99c242719ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.830183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8ad9-8a57-4ac9-9c5f-99c242719ec5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.831688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 192/377] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:40.836636 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8ad9-8a57-4ac9-9c5f-99c242719ec5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:40.854715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.871559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9babbc8c-7abf-405c-8983-d9a5c315609b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:40.872202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9babbc8c-7abf-405c-8983-d9a5c315609b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.872488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9babbc8c-7abf-405c-8983-d9a5c315609b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.872761 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9babbc8c-7abf-405c-8983-d9a5c315609b None admin] PATH_INFO: `/v3/projects/a501f44f388545198539783ea38ef108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.873996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9babbc8c-7abf-405c-8983-d9a5c315609b None admin] RBAC: Authorizing `identity:get_project(project_id=a501f44f388545198539783ea38ef108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:40.877732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9babbc8c-7abf-405c-8983-d9a5c315609b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:40.880027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 186/378] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/projects/a501f44f388545198539783ea38ef108 => generated 344 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:40.881427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:40.881975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:40.881975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:40.906101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:40.907340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:40.910530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:40.985572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8ad9-8a57-4ac9-9c5f-99c242719ec5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:40.999074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 187/379] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:41.000573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:41.076617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:41.077219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:41.077564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:41.077910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:41.078977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:41.079088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:41.080602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:41.097984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 193/380] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 260 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:41.111007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41355d1-d0a6-4213-8326-45fc392c14cd None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:41.149256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:41.150198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:41.150656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:41.151203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] PATH_INFO: `/v3/groups/1d5a8b51c9bd47c1abd94736a02950f8` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:41.158860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] RBAC: Authorizing `identity:get_group(group_id=1d5a8b51c9bd47c1abd94736a02950f8)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:41.160803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:41.162716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 188/381] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:40 2022] GET /identity/v3/groups/1d5a8b51c9bd47c1abd94736a02950f8 => generated 229 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:41.168808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:41.257678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:41.282570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b7b36-6429-4d78-bac0-59e1755cf51d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:41.321721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:41.322396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:41.322738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:41.323042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:41.323786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] RBAC: Adding query filter params (group.id=1d5a8b51c9bd47c1abd94736a02950f8, role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=a501f44f388545198539783ea38ef108) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:41.324095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:41.325575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:41.331800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 194/382] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:41 2022] GET /identity/v3/role_assignments?group.id=1d5a8b51c9bd47c1abd94736a02950f8&scope.project.id=a501f44f388545198539783ea38ef108&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 260 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:41.387633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:41.390856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:41.513230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd2561-6a53-4876-9ba5-5c3fe0dc8dad None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:41.549032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:41.549672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:41.549955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:41.550231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:41.562112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:41.567601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:41.568506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:41.569541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 189/383] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:41.580049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:41.684665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fedaa4f-e066-4afb-a738-33e12f3f5d3e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:41.719918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:41.720529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:41.720815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:41.721093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:41.721765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:41.722073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:41.723496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:41.732777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 195/384] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:41.735301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:41.738409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:41.807766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:41.843978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2663b0-4b1e-48cd-ad40-37a231ccd612 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:41.879416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:41.880054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:41.880335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:41.880609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:41.881728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:41.883213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:41.884942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 190/385] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:41 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:41.890365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:41.997890 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e8d1be-4c20-458e-9c8d-232bb61382c0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:42.032762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c2c00c-3157-44b8-86ac-af2a84601f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:42.033398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c2c00c-3157-44b8-86ac-af2a84601f54 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:42.033679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c2c00c-3157-44b8-86ac-af2a84601f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:42.033952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c2c00c-3157-44b8-86ac-af2a84601f54 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:42.035135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c2c00c-3157-44b8-86ac-af2a84601f54 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:42.036595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c2c00c-3157-44b8-86ac-af2a84601f54 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:42.038422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 196/386] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:41 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:42.044024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:42.147447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf068e59-4522-4150-9363-d4328f3d3737 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:42.177141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:42.183640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf645b8-9566-4a33-8ae8-6a690839e382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:42.184365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf645b8-9566-4a33-8ae8-6a690839e382 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:42.184646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf645b8-9566-4a33-8ae8-6a690839e382 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:42.184920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf645b8-9566-4a33-8ae8-6a690839e382 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec/groups/03f04ef4684c4c58aba9325029f9a258/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:42.187078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf645b8-9566-4a33-8ae8-6a690839e382 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb4febd431cc435cbade69ba4716e3ec, group_id=03f04ef4684c4c58aba9325029f9a258, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:42.188695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf645b8-9566-4a33-8ae8-6a690839e382 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:42.206060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 191/387] 10.222.0.20 () {58 vars in 1442 bytes} [Fri Sep 9 04:25:42 2022] PUT /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec/groups/03f04ef4684c4c58aba9325029f9a258/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:42.639285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75165d6-f0a7-408a-8e3a-b4a905015077 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:42.639757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75165d6-f0a7-408a-8e3a-b4a905015077 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:42.640150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75165d6-f0a7-408a-8e3a-b4a905015077 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:42.641855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 197/388] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:42.647672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:42.647960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:42.648241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:42.675816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:42.677160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:42.757042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 192/389] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:43.122472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75165d6-f0a7-408a-8e3a-b4a905015077 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:43.206254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:43.263844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75165d6-f0a7-408a-8e3a-b4a905015077 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:43.316252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:43.316909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:43.317191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:43.317462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:43.331755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:43.337033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:43.337927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:43.338690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 198/390] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:43.347544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:43.480449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e639738-261e-401d-a9d8-5bc5026861f8 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:43.506599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:43.531552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:43.532180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:43.532465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:43.532741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:43.533418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:43.533724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:43.535131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321441e2-8540-4808-80bf-4ece328aa70d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:43.544721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 193/391] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:43.551097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:43.709672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76aedbe-db0d-4b3f-84d1-47e88295067e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:43.748583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:43.752519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e1e93-82ae-42cf-a1cd-bdb9a5d807b5 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:43.752816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e1e93-82ae-42cf-a1cd-bdb9a5d807b5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:43.753117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e1e93-82ae-42cf-a1cd-bdb9a5d807b5 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:43.754684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 194/392] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:43.757919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:43.758844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:43.759555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:43.759854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:43.760946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:43.761555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:43.761852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:43.762659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:43.762951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:43.764512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 199/393] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:43 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:43.771220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:43.795999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:43.797218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:43.872070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 195/394] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:43.894498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233d92a6-1050-456a-bbd1-ff1870086b06 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:43.935432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:43.936071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:43.936431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:43.936818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:43.938707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:43.940995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:43.942990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 200/395] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:43 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:43.950781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:44.071789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dfb2de-9886-4bd5-bd63-b408d2047fc6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:44.091998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:44.113001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:44.113669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.113949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.114225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec/groups/03f04ef4684c4c58aba9325029f9a258/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.116433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb4febd431cc435cbade69ba4716e3ec, group_id=03f04ef4684c4c58aba9325029f9a258, role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:44.117885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:44.141044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 196/396] 10.222.0.20 () {58 vars in 1442 bytes} [Fri Sep 9 04:25:43 2022] PUT /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec/groups/03f04ef4684c4c58aba9325029f9a258/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:25:44.178308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:44.281733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:44.353093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2639023-9a47-49ba-a936-232394cbe1ea None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:44.394165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3545f3-d10c-457c-b009-cb75b6a6e141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:44.395074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3545f3-d10c-457c-b009-cb75b6a6e141 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.395502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3545f3-d10c-457c-b009-cb75b6a6e141 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.395947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3545f3-d10c-457c-b009-cb75b6a6e141 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.408704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3545f3-d10c-457c-b009-cb75b6a6e141 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0c3545f3-d10c-457c-b009-cb75b6a6e141 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:44.414317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:44.415516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:44.416503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:44.419044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 201/397] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:44.425841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:44.447923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82535dc6-894a-4394-afe9-0c0025bbf606 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.448446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82535dc6-894a-4394-afe9-0c0025bbf606 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.448874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82535dc6-894a-4394-afe9-0c0025bbf606 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.451725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 202/398] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:44.457815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ebabc9-fc7b-4ad9-9041-6a2a7ee1d94b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.458525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ebabc9-fc7b-4ad9-9041-6a2a7ee1d94b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.458985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ebabc9-fc7b-4ad9-9041-6a2a7ee1d94b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.491165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-58ebabc9-fc7b-4ad9-9041-6a2a7ee1d94b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:44.492767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-58ebabc9-fc7b-4ad9-9041-6a2a7ee1d94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:44.546837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68fbf50-386b-4e61-8ead-e9f73f7c8881 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:44.584222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:44.584923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.585270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.585668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.591372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:44.591372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:44.591372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:44.591630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 203/399] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:44.608342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac1a7e7-0a93-4537-be18-2cc5a6d66194 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.608640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac1a7e7-0a93-4537-be18-2cc5a6d66194 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.608833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac1a7e7-0a93-4537-be18-2cc5a6d66194 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.611664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 197/400] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:44.612602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 204/401] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:44 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:44.617435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:44.619084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.619375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.620113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.657790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:44.659432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:44.690726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:44.772733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:44.798437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 205/402] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:44.815098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec2e811-163c-40d6-bf9f-8a44a77fd720 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:44.881598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f69a596-c860-4745-bcec-827d928f2b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:44.882583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f69a596-c860-4745-bcec-827d928f2b10 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.883100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f69a596-c860-4745-bcec-827d928f2b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.883580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f69a596-c860-4745-bcec-827d928f2b10 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.887594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f69a596-c860-4745-bcec-827d928f2b10 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:44.889257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f69a596-c860-4745-bcec-827d928f2b10 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:44.891401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 198/403] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:44 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:44.896786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:44.898819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133fe08e-3703-43f7-83bd-055d50ad778f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.899467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133fe08e-3703-43f7-83bd-055d50ad778f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.899944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133fe08e-3703-43f7-83bd-055d50ad778f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.901525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 199/404] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:44.907385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:44.907906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:44.908377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:44.939361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:44.940883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:45.013385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 200/405] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:45.014102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.036372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1435c034-cbc4-47fc-be9a-bf40fda65f7a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.061392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.080312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.081342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.081819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.082320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.083846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.085545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.088042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 206/406] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:44 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:45.095507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.163013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.168561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1048adb-6501-4f29-a73d-a99d14baff81 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.205933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.206870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.207330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.207781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.218473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:45.223518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e958effb-4594-4afc-9cdd-26d19161d3fa None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:45.223994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:45.225050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:45.226052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:45.238233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 201/407] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:45.239686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.266760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.267531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.267811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.268131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/groups/03f04ef4684c4c58aba9325029f9a258/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.271083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] RBAC: Authorizing `identity:create_grant(project_id=d83b0cd39a3c4abebd2479068e9d13c4, group_id=03f04ef4684c4c58aba9325029f9a258, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.272631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.295014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 207/408] 10.222.0.20 () {58 vars in 1442 bytes} [Fri Sep 9 04:25:45 2022] PUT /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/groups/03f04ef4684c4c58aba9325029f9a258/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:25:45.295014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.382690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.396800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc0e0c5-ed5a-4b9c-a0dc-ec1ff4826206 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.410296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc9b26-ace8-43a6-841f-13d5c98d2541 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.425980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.444819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.446100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.446560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.447016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.449907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.450599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.456637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.456637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.456637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:45.456637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.456637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.463184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 208/409] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:45.464714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:45.468779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:45.470580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:45.471499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:45.472297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:45.475000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 202/410] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:45.476072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.595006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ebb64-bc51-420c-8811-8663ddc869ce None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.602920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790e84e2-94cd-448a-bb0a-e5dc5d6f69b5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.640474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.642085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.642688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.643235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.644911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.645889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.646521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.646800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.647108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.647584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.650117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 203/411] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:45.651167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.671158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:45.677517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:45.678370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:45.679187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 209/412] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:45.679868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.682107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.725246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.773020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19839673-b9f8-4871-a48c-faeedb324d8e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.799723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32074a3-49ed-484b-8a93-76f3a7f78631 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.810867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.811518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.811799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.812277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.812829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:45.813137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.814505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.827535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 204/413] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:45.828545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.840499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.841436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.841974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.842440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.843790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.845636 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.851050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 210/414] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:45.852161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:45.915790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.937890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77df32b-a91c-47a1-a65f-adbd696e0947 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.967355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fbf66e-38be-42bc-8ae2-81a8a5d5c4af None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:45.969308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:45.974320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:45.974956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:45.975297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:45.975579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:45.976256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] RBAC: Adding query filter params (name=member) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:45.976559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:45.977950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:45.987625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 205/415] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:45.988672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:46.009594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.010544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.011148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.011704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] PATH_INFO: `/v3/groups/1d5a8b51c9bd47c1abd94736a02950f8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.013556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] RBAC: Authorizing `identity:get_group(group_id=1d5a8b51c9bd47c1abd94736a02950f8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.015328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.017412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 211/416] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:45 2022] GET /identity/v3/groups/1d5a8b51c9bd47c1abd94736a02950f8 => generated 229 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:46.018380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:46.100766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f2032a-6984-4b68-8b77-f2bff91e3912 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:46.129645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06ae74c-416c-44de-a2e9-de3979db35f9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:46.137871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.138565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.138844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.139165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/groups/03f04ef4684c4c58aba9325029f9a258/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.141716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] RBAC: Authorizing `identity:create_grant(project_id=d83b0cd39a3c4abebd2479068e9d13c4, group_id=03f04ef4684c4c58aba9325029f9a258, role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.143315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.162961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 206/417] 10.222.0.20 () {58 vars in 1442 bytes} [Fri Sep 9 04:25:45 2022] PUT /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4/groups/03f04ef4684c4c58aba9325029f9a258/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:46.162961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:46.187484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:46.192575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.193898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.195315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.195679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.262975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:46.294671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.296258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.388815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e06be-966f-4db7-8cad-e99f8297be43 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:46.434818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.435503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.435780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.436080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] PATH_INFO: `/v3/projects/a501f44f388545198539783ea38ef108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.437322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] RBAC: Authorizing `identity:get_project(project_id=a501f44f388545198539783ea38ef108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.438762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.443910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 207/418] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:46 2022] GET /identity/v3/projects/a501f44f388545198539783ea38ef108 => generated 344 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:46.446649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:46.517179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 212/419] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:46.532333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:46.546775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:46.623255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23889add-a931-4d1d-8626-e08d5dc5aee9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:46.636243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:46.671696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234eb85-fdeb-4aa5-801c-73e2f0bc8a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.672276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234eb85-fdeb-4aa5-801c-73e2f0bc8a8a None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.672547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234eb85-fdeb-4aa5-801c-73e2f0bc8a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.672807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234eb85-fdeb-4aa5-801c-73e2f0bc8a8a None admin] PATH_INFO: `/v3/projects/a501f44f388545198539783ea38ef108/groups/1d5a8b51c9bd47c1abd94736a02950f8/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.674700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f234eb85-fdeb-4aa5-801c-73e2f0bc8a8a None admin] RBAC: Authorizing `identity:create_grant(project_id=a501f44f388545198539783ea38ef108, group_id=1d5a8b51c9bd47c1abd94736a02950f8, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.676268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f234eb85-fdeb-4aa5-801c-73e2f0bc8a8a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.682105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696dcb89-1b1c-4fa5-bad3-9f2f3a0bed87 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:46.691437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 208/420] 10.222.0.20 () {58 vars in 1442 bytes} [Fri Sep 9 04:25:46 2022] PUT /identity/v3/projects/a501f44f388545198539783ea38ef108/groups/1d5a8b51c9bd47c1abd94736a02950f8/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:25:46.692996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376677a-ce85-4f95-aebd-7df7e47638a6 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.693483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376677a-ce85-4f95-aebd-7df7e47638a6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.694031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376677a-ce85-4f95-aebd-7df7e47638a6 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.696063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 209/421] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:46.709935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.710512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.710999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.753565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:46.754536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.755486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.755940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.756384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.758757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:46.762258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.763879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.765654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 213/422] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:46 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:46.771352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:46.891398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 210/423] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:46.914626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602f3e0-d86f-49fa-a6e7-6a433bbf73ce None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:46.967895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:46.968994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:46.969474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:46.969897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:46.971254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:46.978059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:46.980236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 214/424] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:46 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:46.985814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:47.057323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:47.107710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5627ddf0-f5d9-4c5d-8d44-dbe83684b6cd None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:47.148268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:47.149139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:47.149559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:47.149971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:47.150835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=fb4febd431cc435cbade69ba4716e3ec) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:47.151322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:47.153544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:47.160025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 211/425] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:46 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=807749e5ca90423999feeb8a7727599c => generated 611 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:47.202209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:47.212512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:47.341147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e7a75-28d4-467e-8003-5b306611348a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:47.380462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:47.381107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:47.381387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:47.381659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:47.393308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:47.398600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:47.399470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:47.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 215/426] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:47.405900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:47.516185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a04265-42b7-475f-a790-5cce9816798a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:47.542230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:47.551629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:47.552235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:47.552517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:47.552804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:47.553443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:47.553745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:47.555285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d45a9f-129c-4785-9d67-577a940ecdc0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:47.564316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 212/427] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:47.569875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:47.640813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:47.677664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e13249-8475-4fe8-afe7-20d4445415f1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:47.714276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:47.715052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:47.715340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:47.715613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:47.788605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:47.789997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:47.845225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b1d81-b893-47fa-b255-fd033b52ed24 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:47.845507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b1d81-b893-47fa-b255-fd033b52ed24 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:47.845777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b1d81-b893-47fa-b255-fd033b52ed24 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:47.847257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 213/428] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:47.859241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:47.859647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:47.860338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:47.874752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:47.887091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:47.888557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:47.912265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 216/429] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:47.915536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:47.996624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 214/430] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:48.042832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc39e9a7-1d85-415f-8be3-3e9f7e54fab5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:48.081522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:48.082164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.082443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.082716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] PATH_INFO: `/v3/projects/fb4febd431cc435cbade69ba4716e3ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.086782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] RBAC: Authorizing `identity:get_project(project_id=fb4febd431cc435cbade69ba4716e3ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:48.088282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:48.090730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 217/431] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:47 2022] GET /identity/v3/projects/fb4febd431cc435cbade69ba4716e3ec => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:48.096052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:48.219206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faf56a4-021f-4cd6-b45b-83f485b5eb67 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:48.243300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:48.277273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:48.277955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.278236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.278512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.282085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:48.283780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:48.285886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 215/432] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:48 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:48.292002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:48.307552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:48.350181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:48.430021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaa0a5e-b545-429f-aa83-15003f2ba5e3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:48.444369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b4572e-a573-4425-8e1a-cd673c32b453 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:48.469816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:48.471215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.471687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.472437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.473382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=fb4febd431cc435cbade69ba4716e3ec) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:48.473909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:48.475601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:48.483497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 218/433] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:48 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=fb4febd431cc435cbade69ba4716e3ec&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 611 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:48.487109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68530492-7701-4e62-8274-235d43be9e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:48.487832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68530492-7701-4e62-8274-235d43be9e95 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.488180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68530492-7701-4e62-8274-235d43be9e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.488545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68530492-7701-4e62-8274-235d43be9e95 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.509325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68530492-7701-4e62-8274-235d43be9e95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-68530492-7701-4e62-8274-235d43be9e95 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:48.514682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:48.515696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 09 04:25:48.516999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:48.521132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 216/434] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:48.522590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:48.639160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:48.643695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec58f129-59dd-42c2-b917-c07296ac609b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:48.680944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:48.681558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.681839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.682120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.683082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:48.683387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:48.685295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:48.693922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 219/435] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:25:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:48.699857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ae3b3-5414-4c76-92f7-a8de689aa34c None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.700147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ae3b3-5414-4c76-92f7-a8de689aa34c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.700426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ae3b3-5414-4c76-92f7-a8de689aa34c None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.701075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:48.701860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 217/436] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:48.707224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.707515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.707795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.729031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:48.730351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:48.790773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:48.823603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eecad04-ab77-49e9-8450-1e78ddc3b7c4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:48.826209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 218/437] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:48.868404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:48.869401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:48.869685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:48.869964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:48.905086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:48.946871 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:48.948345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:49.053101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 220/438] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:49.054560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.096696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.157628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5814fecb-9936-478f-8bcb-adb6f8abdb60 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.162000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.196085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.196736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.197014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.197288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.198445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:49.200502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:49.202132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 219/439] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:49.207496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.210481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f4d0cb-41fc-4a9c-b3dc-5623be0f9196 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.250323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.250966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.251432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.251548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.254408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.266561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:49.271983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:49.272877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:49.274591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 221/440] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:49.279268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.313036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac800096-8fa0-4c72-8cba-80f6e3c321f0 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.345500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7106be5-0238-4a0e-b31a-56fd3c07d37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.346126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7106be5-0238-4a0e-b31a-56fd3c07d37a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.346406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7106be5-0238-4a0e-b31a-56fd3c07d37a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.346682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7106be5-0238-4a0e-b31a-56fd3c07d37a None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.347829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7106be5-0238-4a0e-b31a-56fd3c07d37a None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:49.349270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7106be5-0238-4a0e-b31a-56fd3c07d37a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:49.350824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 220/441] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:49.351947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ccb060-3b12-42e2-addd-6bab73122731 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.352233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ccb060-3b12-42e2-addd-6bab73122731 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.352511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ccb060-3b12-42e2-addd-6bab73122731 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.353707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 221/442] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:49.358503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ccb060-3b12-42e2-addd-6bab73122731 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.377301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bd8f44-6104-43ce-a221-5067a08530b7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.397024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.412290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.412832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.413113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.413389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.414049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:49.414359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:49.415771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f4ba66-1ce0-43b8-8557-043603365335 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:49.423672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 222/443] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:49.424801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.425087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.425362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.445884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:49.447114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:49.462885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ccb060-3b12-42e2-addd-6bab73122731 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.475977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.498128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.498740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.499051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.499333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.500063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=807749e5ca90423999feeb8a7727599c, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:49.500385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:49.501851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:49.507457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 222/444] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=807749e5ca90423999feeb8a7727599c => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:49.508223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.515087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 223/445] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:49.623538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64524b75-9a93-428e-a7be-566a6ff9fedf None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.661130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.662198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.662746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.663247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.721551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.748364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:49.749725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:49.795994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.863229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 223/446] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:49.869345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:49.917066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20da9da-7341-4845-aac7-4864e6cd6f8c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.930257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.951024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:49.954158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:49.954778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:49.955091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:49.955365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:49.965286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:49.970104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:49.971007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:49.971814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 224/447] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:49.976361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a51683-6464-48f2-a3f1-d85439134252 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:49.976984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.010256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.010977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.011259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.011526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] PATH_INFO: `/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.012637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] RBAC: Authorizing `identity:get_project(project_id=d83b0cd39a3c4abebd2479068e9d13c4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.014087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.015789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 224/448] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/projects/d83b0cd39a3c4abebd2479068e9d13c4 => generated 302 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:50.022030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.081617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee19e334-58be-4249-9af3-e85c2ffad35e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.115940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.116539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.116815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.117084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.117742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:50.118042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.119440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.127570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 225/449] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:25:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:50.128816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ce35f-cb3a-43b0-af62-6d31117e280c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.133150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.162414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.163071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.163351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.163634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] PATH_INFO: `/v3/groups/03f04ef4684c4c58aba9325029f9a258` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.164750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] RBAC: Authorizing `identity:get_group(group_id=03f04ef4684c4c58aba9325029f9a258)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.166209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.166695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:50.168533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 225/450] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:50 2022] GET /identity/v3/groups/03f04ef4684c4c58aba9325029f9a258 => generated 226 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:50.174158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.188290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:50.234967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e360db7-9dae-4b3c-893f-81ef5e3e2006 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.267260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.267869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.268149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.268430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.273067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec7c8ef-c6b3-4c3d-b12a-8166993082cd None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.305386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.305975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.306251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.306518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.307275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] RBAC: Adding query filter params (group.id=03f04ef4684c4c58aba9325029f9a258, role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:50.307574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.309809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.315377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 226/451] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:50 2022] GET /identity/v3/role_assignments?group.id=03f04ef4684c4c58aba9325029f9a258&scope.project.id=d83b0cd39a3c4abebd2479068e9d13c4&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 611 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:50.336652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.338147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.373702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:50.450213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 226/452] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:25:50.455909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.534599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:50.559962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87938559-22ec-4859-9513-b123e25d69ed None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.593519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.594142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.594417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.594687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] PATH_INFO: `/v3/projects/a501f44f388545198539783ea38ef108` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.595830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] RBAC: Authorizing `identity:get_project(project_id=a501f44f388545198539783ea38ef108)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.597263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.598866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 227/453] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:25:50 2022] GET /identity/v3/projects/a501f44f388545198539783ea38ef108 => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:50.604436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.704092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa727148-2084-4b66-8fa1-3e439fcb1b49 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.737262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1097d935-093d-4e33-835e-cb353003b906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.737876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1097d935-093d-4e33-835e-cb353003b906 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.738155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1097d935-093d-4e33-835e-cb353003b906 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.738428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1097d935-093d-4e33-835e-cb353003b906 None admin] PATH_INFO: `/v3/groups/1d5a8b51c9bd47c1abd94736a02950f8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.739515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1097d935-093d-4e33-835e-cb353003b906 None admin] RBAC: Authorizing `identity:get_group(group_id=1d5a8b51c9bd47c1abd94736a02950f8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.740947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1097d935-093d-4e33-835e-cb353003b906 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.742481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 227/454] 10.222.0.20 () {56 vars in 1180 bytes} [Fri Sep 9 04:25:50 2022] GET /identity/v3/groups/1d5a8b51c9bd47c1abd94736a02950f8 => generated 229 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:50.748372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:50.844256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8a6a04-cbb4-4a1b-a09b-0511035a022d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:50.875992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:50.876576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:50.876854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:50.877124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:50.877850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] RBAC: Adding query filter params (group.id=1d5a8b51c9bd47c1abd94736a02950f8, role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=a501f44f388545198539783ea38ef108) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:50.878195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:50.879642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478126d5-2e22-4d15-ab16-fba2e5500be2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:50.885068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 228/455] 10.222.0.20 () {56 vars in 1376 bytes} [Fri Sep 9 04:25:50 2022] GET /identity/v3/role_assignments?group.id=1d5a8b51c9bd47c1abd94736a02950f8&scope.project.id=a501f44f388545198539783ea38ef108&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 611 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:53.544013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea091b1-6d89-4121-87b7-a3f1fab61f11 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:53.545027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea091b1-6d89-4121-87b7-a3f1fab61f11 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:53.545513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea091b1-6d89-4121-87b7-a3f1fab61f11 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:53.547305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 228/456] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:53.552242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8128b5e-b638-4e27-b6ac-7cc242b6bb75 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:53.553047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8128b5e-b638-4e27-b6ac-7cc242b6bb75 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:53.553537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8128b5e-b638-4e27-b6ac-7cc242b6bb75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:53.583908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b8128b5e-b638-4e27-b6ac-7cc242b6bb75 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:53.585385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b8128b5e-b638-4e27-b6ac-7cc242b6bb75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:53.587707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:53.668371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 229/457] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:53.902835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e4815c-9470-4c6e-9c60-a956ba3e906b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:53.903380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e4815c-9470-4c6e-9c60-a956ba3e906b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:53.903795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e4815c-9470-4c6e-9c60-a956ba3e906b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:53.905471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 229/458] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:53.911054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bc6830-d00f-4c3a-ad0c-c6569c8c943c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:53.911342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bc6830-d00f-4c3a-ad0c-c6569c8c943c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:53.911609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bc6830-d00f-4c3a-ad0c-c6569c8c943c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:53.917300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42010c76-935d-4f8c-82cb-66e56384615d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:53.917848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42010c76-935d-4f8c-82cb-66e56384615d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:53.918266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42010c76-935d-4f8c-82cb-66e56384615d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:53.920062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 230/459] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:53.932504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:53.933056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:53.934047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:53.936054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-41bc6830-d00f-4c3a-ad0c-c6569c8c943c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:53.937523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-41bc6830-d00f-4c3a-ad0c-c6569c8c943c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:53.943855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:53.961627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:53.962860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:54.060002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 230/460] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:54.061630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8607f4a-3631-4959-ad93-8db1e2898fbe None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.062101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8607f4a-3631-4959-ad93-8db1e2898fbe None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.062577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8607f4a-3631-4959-ad93-8db1e2898fbe None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.064051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 231/461] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:54.069180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.069663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.070129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.077942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 231/462] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:54.097877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.100332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:54.101529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:54.220156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:54.232605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 232/463] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:54.254894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116dfa44-9cfc-487b-8243-10223bfbee54 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.304127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:54.311326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.312177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.312650 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.313136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.323492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.325752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:54.340597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:54.341488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:54.342317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:54.342317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.342317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:54.344276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 232/464] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:54.350313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.374811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.482515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06f19a3-d297-4114-a211-5da2b0acc5e6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.505395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42986d8-0b7d-4196-b735-96b18ee983c2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.521547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.522491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.522991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.523653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.524598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:54.525108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.526670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.543730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.544079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 233/465] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:54.544191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.544606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.544606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.545779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.556259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.557976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:54.574456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:54.575457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:54.577036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:54.577036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.577036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:54.577036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 233/466] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:54.577036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.614789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:54.616614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:54.660679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692326d-e2a5-4f50-9244-c634af29b418 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.688960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb2ccc9-1719-4ecd-af49-1a010db49013 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.693896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.694524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.694802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.695107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.705692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.707180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:54.721712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:54.722615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:54.724269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:54.724269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.724269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:54.724269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 234/467] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:54.724269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.726666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.727431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.728018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.728279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.728928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.730420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.754515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 234/468] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:54 2022] POST /identity/v3/users => generated 246 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:54.755458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.856363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37ec6b-a106-4718-a6e2-bcc69d643271 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.872851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:54.875481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:54.888710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b09669-e75c-4542-87ac-148ccb24d25f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:54.909375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.910207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.910571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.910974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.925945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.927768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.930780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:54.931533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:54.931888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:54.932237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:54.933280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:54.933662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:54.935325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:54.950163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:54.951424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:54.952268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:54.952268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:54.952268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:54.952268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 235/469] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:54.952772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:54.956362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 235/470] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:54.958158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:55.088704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072f04da-5195-4291-be3a-82e0aaa7d366 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:55.092392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222defcb-0ab5-45f3-9155-f1b3c983fd57 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:55.127226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:55.127829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.128104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.128372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.129006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:55.129207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:55.129316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:55.129815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.130091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.130359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.130673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:55.131025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:55.131329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:55.132669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a728a-6724-4665-a0d2-c969804b768c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:55.143840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 236/471] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:55.148867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:55.148994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 236/472] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:25:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:55.148994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e5451-80e5-4316-9045-7ed1230db3ae None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.148994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e5451-80e5-4316-9045-7ed1230db3ae None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.148994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e5451-80e5-4316-9045-7ed1230db3ae None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.151884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 237/473] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:55 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:55.153971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9e5451-80e5-4316-9045-7ed1230db3ae None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:55.172041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:55.174983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:55.257754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d32fd06-050a-4ad2-9ac9-0d55fa136c67 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:55.274542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9e5451-80e5-4316-9045-7ed1230db3ae None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:55.295924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:55.296530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.296808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.297074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.297722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:55.299179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:55.314145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5478067a-2cb7-4c9d-a650-8f020b90de40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:55.314800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5478067a-2cb7-4c9d-a650-8f020b90de40 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.315085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5478067a-2cb7-4c9d-a650-8f020b90de40 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.315355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5478067a-2cb7-4c9d-a650-8f020b90de40 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.316280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5478067a-2cb7-4c9d-a650-8f020b90de40 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:55.317797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5478067a-2cb7-4c9d-a650-8f020b90de40 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:55.330979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 237/474] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:55 2022] POST /identity/v3/users => generated 248 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:55.330979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:55.345524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 238/475] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:55 2022] POST /identity/v3/users => generated 249 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:55.346746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.347052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.347381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.372770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:55.374055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:55.451960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:55.459858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 239/476] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:55.500455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:55.536588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa641b26-16b3-4634-a5ee-ba3ceb77c1b5 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:55.577496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29fd083-d806-462a-b27e-796629467219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:55.578511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29fd083-d806-462a-b27e-796629467219 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.579030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29fd083-d806-462a-b27e-796629467219 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.579508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29fd083-d806-462a-b27e-796629467219 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.580455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29fd083-d806-462a-b27e-796629467219 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:55.582160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29fd083-d806-462a-b27e-796629467219 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:55.610293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 238/477] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:25:55 2022] POST /identity/v3/users => generated 248 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:25:55.764065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:55.873157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16593efc-9d7f-4090-86f1-7997b0f4bf84 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:55.907962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:55.908570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:55.908848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:55.909120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:55.909794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:55.911194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:55.919892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.sql.core [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 09 04:25:55.919892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 09 04:25:55.919892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [parameters: {'id': '3aadbe65c7e045fdb26d4479097b56be', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 09 04:25:55.919892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Sep 09 04:25:55.936639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return ref.to_dict() Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 09 04:25:55.937556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application yield resource Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.rollback() Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.force_reraise() Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise self.value Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application session.commit() Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 09 04:25:55.938438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._prepare_impl() Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.flush() Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._flush(objects) Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application compat.raise_( Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application flush_context.execute() Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rec.execute(self) Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 09 04:25:55.939357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 09 04:25:55.940214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 09 04:25:55.941007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [parameters: {'id': '3aadbe65c7e045fdb26d4479097b56be', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:55.941787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 09 04:25:55.942634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:55.945543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 240/478] 10.222.0.20 () {60 vars in 1131 bytes} [Fri Sep 9 04:25:55 2022] POST /identity/v3/roles => generated 144 bytes in 175 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:55.947628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29fd083-d806-462a-b27e-796629467219 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:56.059164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29fd083-d806-462a-b27e-796629467219 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:56.101972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46631362-be1f-470e-b90d-38c3290d5fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:56.103046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46631362-be1f-470e-b90d-38c3290d5fba None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:56.103546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46631362-be1f-470e-b90d-38c3290d5fba None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:56.104016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46631362-be1f-470e-b90d-38c3290d5fba None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:56.106204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:56.121591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46631362-be1f-470e-b90d-38c3290d5fba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-46631362-be1f-470e-b90d-38c3290d5fba None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:56.129559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:56.130425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:56.131258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 239/479] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:25:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:56.138241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:56.203634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:56.246280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac0a36-11fd-460c-a9eb-51a2a4b8530f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:56.284346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:56.284955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:56.285230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:56.285496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:56.286150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:56.286449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:56.287847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16cad9-2a99-4c7c-9099-ba1f2032c9ce None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:56.296474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 241/480] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:25:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:57.265045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fb77f5-7714-4c0f-a2be-e5bf065b243f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:57.265492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fb77f5-7714-4c0f-a2be-e5bf065b243f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:57.265582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fb77f5-7714-4c0f-a2be-e5bf065b243f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:57.267299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 240/481] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:57.272104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:57.272499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:57.272676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:57.299047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:57.300225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:57.398166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 242/482] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:57.767157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fb77f5-7714-4c0f-a2be-e5bf065b243f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:57.910555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fb77f5-7714-4c0f-a2be-e5bf065b243f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:57.957039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:57.957681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:57.957962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:57.958328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:57.974793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:57.980103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:57.980950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:57.981721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 241/483] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:25:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:57.986948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:58.061818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:58.119459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4f21c7-507b-4f96-890f-d293c9f0eec8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:58.176128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:58.176748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.177009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.177272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.177945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:58.178247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:58.183538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b462ada-081e-4540-9dd6-2a931910fa64 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:58.194018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 243/484] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:25:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:58.200261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:58.284369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:58.301070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc93c66-5d4e-4e70-a989-8da46eb29d20 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.301659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc93c66-5d4e-4e70-a989-8da46eb29d20 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.301659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc93c66-5d4e-4e70-a989-8da46eb29d20 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.307077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 244/485] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:25:58.309010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782b05c4-7ee3-40f1-8f55-7b9324921ad4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.309295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782b05c4-7ee3-40f1-8f55-7b9324921ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.309571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782b05c4-7ee3-40f1-8f55-7b9324921ad4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.338570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-782b05c4-7ee3-40f1-8f55-7b9324921ad4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:58.340235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-782b05c4-7ee3-40f1-8f55-7b9324921ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:58.361352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848f5c11-524f-45d2-855f-adaed58db277 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:58.404163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:58.405498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.406032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.406483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.416284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 245/486] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:58.420732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:58.422477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:58.424840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc430543-f224-4610-a073-9fb7160741c1 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.425136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc430543-f224-4610-a073-9fb7160741c1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.425420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc430543-f224-4610-a073-9fb7160741c1 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.427396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 246/487] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:58 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:58.433895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c39fddd-cc3c-482d-974a-72ac93014195 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.434457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c39fddd-cc3c-482d-974a-72ac93014195 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.434962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c39fddd-cc3c-482d-974a-72ac93014195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:58.441597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:58.442723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:58.444287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:58.444287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:58.444287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:58.444287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 242/488] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:58.450602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:58.483581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7c39fddd-cc3c-482d-974a-72ac93014195 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:58.484779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7c39fddd-cc3c-482d-974a-72ac93014195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:58.563026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:58.599361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 247/489] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:58.601053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b462c-2736-4e9e-91be-15fbd0212b9b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.601336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b462c-2736-4e9e-91be-15fbd0212b9b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.601754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b462c-2736-4e9e-91be-15fbd0212b9b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.603419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 248/490] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:58.614116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80098ebd-1f73-41cf-8453-8df6df4e7453 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:58.617050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.617335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.617615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.628167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:58.651380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:58.653093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:58.654304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:58.655443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.655905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.656355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.657203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:58.657710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:58.659470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:58.680270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 243/491] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:25:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:58.682185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:58.758751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 249/492] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:25:58.759861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:58.805509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e592a63-987f-4da1-b057-06ff1ef0fd26 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:58.847589 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:58.848311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.848665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.849048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:58.858097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:58.860546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:25:58.878555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:58.879673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:25:58.880766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:58.887054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 244/493] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:25:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:25:58.888820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:58.921815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:58.941014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab988db-abaa-49b4-8926-f5c9d38b3ca2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:58.955134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:58.993036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:58.994345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:58.994704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:58.995230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.005895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd55633-27f3-49b7-a56b-f51129492d0f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.007957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:59.014854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:59.015826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:59.016616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 250/494] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:25:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:59.018246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.041807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.042578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.042971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.043343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.044182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:59.044587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.046122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.093088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 245/495] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:25:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:25:59.094102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.154578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684569f-8613-4f19-99d0-5fa4568cf257 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.182946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.191586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.192317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.192592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.192860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.200620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3937988-7163-4586-93d9-048062f1c0d1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.203754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:59.208933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:59.209813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:59.210604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 251/496] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:59.211264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.240307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.240503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.240916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.241308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.241477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.242187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:59.242790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.244305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.252410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 246/497] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:59.253098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.316643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d273a6-9a8b-4813-b0f8-073e7d6e1689 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.348757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.349352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.349631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.349770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149dcd7-edd9-4735-bae2-979e4ad5d747 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.350190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.382106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc70172-927a-45dd-8de8-f7fcef397319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.382725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc70172-927a-45dd-8de8-f7fcef397319 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.383037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc70172-927a-45dd-8de8-f7fcef397319 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.383323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc70172-927a-45dd-8de8-f7fcef397319 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.393494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc70172-927a-45dd-8de8-f7fcef397319 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-dbc70172-927a-45dd-8de8-f7fcef397319 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:25:59.398688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:59.399591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:59.400516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 247/498] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:59.401321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c98a56-3b65-4a9d-b96a-7e844bf136e4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.401496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c98a56-3b65-4a9d-b96a-7e844bf136e4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.402306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c98a56-3b65-4a9d-b96a-7e844bf136e4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.403820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 248/499] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:25:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:25:59.404517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c98a56-3b65-4a9d-b96a-7e844bf136e4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.413154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.422026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.423589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.449653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.507198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c98a56-3b65-4a9d-b96a-7e844bf136e4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.531727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 252/500] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:25:59.534031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.555514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.556183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.556460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.556734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.557395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:59.558040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.559597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.568547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 249/501] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:59.569335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.631578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e0f5e-33a4-41ef-8106-de12c9c6e2e7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.633524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.658901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.667657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b4f67-e679-48c7-b072-f01e2d49b5ca None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.667960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.668595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.668875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.669145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.679305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.680743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-feed0be4-b016-45e5-9947-11c6f2c3ba8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:59.695378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:59.696367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:59.697244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:59.697244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:59.697244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:59.697244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 253/502] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:59.697761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.698046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.698317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.708267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.708870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.709150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.709422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.710085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:25:59.710389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.711792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.718422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:25:59.719629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:25:59.720251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 250/503] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:25:59.721023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.785499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 254/504] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:25:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:25:59.786461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.831743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a8a985-1cfb-48e1-9309-91c9e925e32e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.861106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.868087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.868739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.869012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.869285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.880976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.882425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:59.896665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:59.897546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:59.898400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:59.898400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:59.898400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:25:59.898400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 251/505] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:59.899106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:25:59.918408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af10a4dd-101d-4b2d-b9a4-2c453877ec13 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:25:59.928403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:25:59.958560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:25:59.959302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:25:59.959650 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:25:59.959990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:25:59.970462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:25:59.971941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:25:59.987618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:25:59.988503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:25:59.989329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:25:59.989329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:25:59.989329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:25:59.993484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 255/506] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:25:59.994099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.002952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880462e-4f99-40bd-80d5-2e0956c2a2dc None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.038014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.038714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.039101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.039446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.040169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.040571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.042013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.058552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 252/507] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.058552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.098355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.102810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83d9e56-7528-46f2-bf3b-6c5849491335 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.136829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.137452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.137732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.138007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.149101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.150549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.155785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90198af-2291-409d-bb6b-9839c28e8b8a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:00.165088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:00.166167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:00.167232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:00.167232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:00.167232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:00.167232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 256/508] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:25:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:00.167232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.185197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.193062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.193668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.193938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.194207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.194845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.195177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.196559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.208168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 253/509] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.208954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.275957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca27094-f752-4017-8f6e-9e1db0af0e76 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.305638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07608d1-c97b-4581-9154-24db7ebd43fb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.307143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.310237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.310867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.311180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.311449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.312077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.312375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.313772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.325503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 257/510] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.326301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.340565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.341190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.341474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.341752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.352642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.354102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:00.368556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:00.369631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:00.371490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:00.371490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:00.371490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:00.371490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 254/511] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:00.371490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.397112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.427646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b467dd2e-4529-49bb-ab73-d4946d7a5fbc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.462068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da711f15-379f-49da-a804-f04a2510617f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.462630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da711f15-379f-49da-a804-f04a2510617f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.462912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da711f15-379f-49da-a804-f04a2510617f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.463216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da711f15-379f-49da-a804-f04a2510617f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.470167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da711f15-379f-49da-a804-f04a2510617f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.471660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da711f15-379f-49da-a804-f04a2510617f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.472400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfca377-d296-4f93-a57a-262c6a75d1c8 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-da711f15-379f-49da-a804-f04a2510617f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:00.485342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:00.486423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 258/512] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/users/neutron => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:00.487372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da711f15-379f-49da-a804-f04a2510617f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.507296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.507901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.508181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.508454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.509096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.509400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.510770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.526977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 255/513] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.526977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.537815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.592409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da711f15-379f-49da-a804-f04a2510617f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.620959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.622151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4786e7e-75d9-419d-8729-acd2599955d6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.626033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.626655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.626958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.627237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.633907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.635391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:00.647251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:00.648070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:00.648905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 259/514] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:00.649376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.657967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.658592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.658872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.659187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.666090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.667636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:00.678755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:00.679672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 256/515] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/users/glance => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:00.680624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.751937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82915fb2-f814-43e4-8927-5b58c483bb58 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.753727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.779958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e06afc-dde2-456b-be0f-51ad4c7a4483 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.784817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.785417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.785698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.785972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.786606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.786913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.788299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.799710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 260/516] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.800504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.814104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.814706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.815014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.815295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.816019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.816327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.817748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.834534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 257/517] 10.222.0.20 () {56 vars in 1139 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.835418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.855570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.903120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fa4ee9-8a65-4e75-9114-53ab026a4979 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.934310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9cef6-8925-4b61-89ec-b16963e7b008 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:00.937162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.937786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.938066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.938339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.946287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.947769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:00.959167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:00.960230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 261/518] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/users/cinder => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:00.961462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:00.965212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:00.970514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:00.971154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:00.971433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:00.971704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:00.972405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:00.972709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:00.974159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:00.992437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 258/519] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:00.993205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.064739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c4d083-2363-4380-93d5-0ac9856b1196 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.077628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.091413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba9f9a-45ad-46a2-a15a-7e5db6cf72be None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.099486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.100097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.100382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.100660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.101392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:01.101708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.103181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.123997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.124600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.124882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.125156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.125806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.127257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.139675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 262/520] 10.222.0.20 () {56 vars in 1137 bytes} [Fri Sep 9 04:26:00 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:01.140496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.162965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 259/521] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:26:00 2022] POST /identity/v3/users => generated 247 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:01.162965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.214913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.259403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44240afd-c1b8-4917-ae73-deb4450ee768 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.278462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bd66c-2005-4a65-bfd6-e082aa373946 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.296115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.296739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.297019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.297296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.315690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.316304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.316607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.319126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.319126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:01.319126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.319478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.338383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 260/522] 10.222.0.20 () {56 vars in 1137 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:01.339673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.347722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.382990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.384471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.450100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff51e1a-e2e7-4948-b0a4-5adf0c5a0075 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.457794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.486009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.486665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.486961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.487242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.487964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=0dd190c922964d51af263442650b427d) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:01.488275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.489714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.502971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 261/523] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/role_assignments?user.id=0dd190c922964d51af263442650b427d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:01.502971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.503330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 263/524] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:01.503330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.588608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.629451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e499635-b01b-4192-b6ef-f19868bd7515 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.643327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b73f9f-a8ff-43ab-9d6d-022909b96e65 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:01.676568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.677472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.677782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.678501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.683279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:01.684101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:01.684478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:01.684847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:01.734911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.765027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.766594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.767077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:01.768804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:01.840001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:01.887421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 264/525] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:01.888489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.897664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 262/526] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:01.898582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:01.986968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.006296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad6c9e-2e76-4be9-a7e6-2f228e0e36c7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.020920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051e74f-4f49-40fd-9845-5d01522d9423 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.053394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.054023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.054298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.054571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.056533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.057154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.057429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.058471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.068127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.069759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.069964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.071302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:02.085912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:02.088044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:02.090154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:02.090154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:02.090154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:02.090376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:02.091692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:02.092746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:02.092746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:02.092746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:02.092969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 263/527] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:02.092969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.093118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 265/528] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:02.103173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.131664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.207417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b2972-debb-49c9-aef3-94d012fbbe85 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.218176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1142fb58-43cb-4c9a-882e-daa3f02975b1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.238026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.255143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.255990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.256921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.257361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.258107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.258899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.259292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.259660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.260399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:02.260799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.262528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.274471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.275912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 266/529] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:02.276349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.277452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:02.295683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:02.296817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:02.297850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:02.297850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:02.297850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:02.300069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 264/530] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:02.301371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.372678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.393489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975ff37b-15c4-4ba5-b7a1-3f4c090f773f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.434279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.435124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.435501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.436067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.437245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:02.437654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.439483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.444785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed7cd8a-3df7-4fbc-8287-ff21cd84cea4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.454252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 267/531] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:02.455557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.485196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.485874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.486242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.486584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.493476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.495034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:02.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:02.508222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:02.509090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 265/532] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:02.509541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.546093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.585403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42dd33f-555b-44a3-a9e2-a91f2b98d93f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.625401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10091d3-b385-4f20-a230-a47fcfdf2b2d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.628752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.629463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.629836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.630204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.630993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:02.631400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.632920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.646338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 268/533] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:02.647415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.650251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.671556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.673163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.673614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.674290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.684333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.686206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:02.699871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:02.700899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:02.701933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:02.703591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 266/534] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:02.704541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.779645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9843d-605c-4942-bea3-f9461bb1be0a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.805240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.816288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db5eed-f73c-470a-a026-a500401ca550 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:02.824125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.825090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.825484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.825851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.826897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:02.827062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.828873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.848105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 269/535] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:02.849234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.861461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:02.862786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:02.863507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:02.863961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:02.873949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:02.876515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:02.895035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:02.896174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:02.897215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:02.899981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 267/536] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:02.901192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:02.946276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:02.966546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a0258e-f92f-45ba-b58c-89af324c08fa None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:03.004855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:03.005471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.005752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.006028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.006727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:03.007096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:03.008540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:03.025230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 270/537] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:03.026131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:03.037736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b746d0-ffde-4f3a-9a89-65fc4b87ccd6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:03.081760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:03.082411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.082695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.083005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.083741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=a8add98306764181b5ea1e8a1190bbde) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:03.084049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:03.085547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:03.099393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 268/538] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:02 2022] GET /identity/v3/role_assignments?user.id=a8add98306764181b5ea1e8a1190bbde&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:03.100392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:03.149590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27213a3a-35f1-48d2-8f42-ff4ed00884e2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:03.164410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:03.183331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:03.211192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:03.211823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.212106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.212377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.213077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:03.213383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:03.215440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:03.252992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 271/539] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:03.271056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:03.292707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709bdf66-55b9-452f-9d59-3d9e900ec830 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:03.336336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:03.337167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.337275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.337532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.338861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=4c501a6bee5b4f3a8973f727f9ef10cb) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:03.339929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:03.341434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486bc9c-759f-4e3a-869e-c88145d66cef None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:03.350000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 269/540] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:03 2022] GET /identity/v3/role_assignments?user.id=4c501a6bee5b4f3a8973f727f9ef10cb&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:03.466525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d71938-8b4c-4cfe-9149-8ff4bb8e504c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:03.467926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d067966-32aa-44fe-831c-25190cd4f7ab None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.468321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d067966-32aa-44fe-831c-25190cd4f7ab None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.468698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d067966-32aa-44fe-831c-25190cd4f7ab None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.470608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 270/541] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:03.476628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf69aa15-577e-49b9-8d32-5273bae21945 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.477241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf69aa15-577e-49b9-8d32-5273bae21945 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.477740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf69aa15-577e-49b9-8d32-5273bae21945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.509537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-cf69aa15-577e-49b9-8d32-5273bae21945 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:03.511658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cf69aa15-577e-49b9-8d32-5273bae21945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:03.523427 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:03.524415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.524915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.525389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.526358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=fa7372a6f3ba4f3786becc1389147d2d) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:03.527483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:03.529186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c502af9-be84-49e2-b013-4f9400f52d3f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:03.542295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 272/542] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:03 2022] GET /identity/v3/role_assignments?user.id=fa7372a6f3ba4f3786becc1389147d2d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:03.654529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 271/543] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:26:03.763226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b5926-d872-4cd7-8cb6-b8eb5cd5a5af None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.763600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b5926-d872-4cd7-8cb6-b8eb5cd5a5af None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.763989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b5926-d872-4cd7-8cb6-b8eb5cd5a5af None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.765755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 273/544] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:03.771555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:03.771839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:03.772112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:03.793159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:03.794382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:03.803191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:03.869770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 272/545] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:26:04.019955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b5926-d872-4cd7-8cb6-b8eb5cd5a5af None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.072833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:04.130562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b5926-d872-4cd7-8cb6-b8eb5cd5a5af None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.165387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.166073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.166358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.166683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.180777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:04.186254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:04.187228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:04.188009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 274/546] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:04.193437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.257220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.326246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c1dd6c-1400-4966-ace8-ead908d15136 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.390494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.391143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.391425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.391703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.392370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:04.392674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:04.394083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:04.399428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd70d5aa-319b-4c63-b8b3-4c4d6d1ef2bf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.415311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:04.417071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 273/547] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:04.423397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.427884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:04.467183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.468898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.469179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.469453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.489192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:04.502190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:04.503117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:04.503907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 275/548] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:04.509609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.609938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1985a930-e035-4d2d-a9cb-14ade6c21f27 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.618548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0526e458-093e-47a4-b9b7-4fb15641b0f1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.654630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.655323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.655612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.655888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.656565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:04.656872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:04.658252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:04.666897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 276/549] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:04.672339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.683400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.684838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.685496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.685966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.703286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:04.706643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:04.708133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:04.732654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:04.733765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:04.734657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:04.734657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:04.734657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:04.739343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 274/550] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:04.744683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.777710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab4dad-c34b-4cf0-8c85-dab29d4014a9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.814349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.815154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.815443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.815717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.826962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:04.828432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:04.843363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:04.844343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:04.845365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:04.845365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:04.845365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:04.845365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 277/551] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:04.850696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:04.888329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:04.935399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:04.953812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f8af-cae9-4ac8-8260-3646b18813eb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.966189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704eb037-bf5f-41e9-a2b4-8fa112f7af26 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:04.994565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:04.995225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:04.995550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:04.995826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:04.996474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:04.996778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:04.998168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.020280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 275/552] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:05.027875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.031017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.031979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.032465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.032964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.033860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:05.034417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.036051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.048541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 278/553] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:05.060274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.183521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975fa44-0b0f-416a-a71f-7424178217c5 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.206039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c7adba-b7b3-4185-9f61-143ee31c5947 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.224791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:05.264530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.264530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.264530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.264530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.272146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.273597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.278642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.279309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.279589 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.279859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.283241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:05.296419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.297885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:05.301534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:05.302442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:05.303380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:05.310100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 279/554] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:05.311483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:05.324972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:05.325997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:05.328445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:05.333921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 276/555] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:05.334788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.440893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82abeb5-7c1e-4fe8-ab8b-1196df73e636 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.480416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.481213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.481569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.481847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.482588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:05.482893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.484475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.505099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 280/556] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:05.511152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.583371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:05.592999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8076cb95-c962-4f17-8933-c0ea014edc44 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.626576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146bdfbf-390d-4958-8733-21185d780471 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.663954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.664586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.664867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.665144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.676932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.678394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.687829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.688815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.689314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.690063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.690389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:05.690698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.692268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0fa90f-58d8-490b-b63c-646ffbd042bb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:05.694284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:05.696671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:05.699688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:05.699688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:05.699688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:05.699688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 281/557] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:05.702985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.718839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 277/558] 10.222.0.20 () {56 vars in 1135 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:05.720494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2bab29-6916-4456-a4e5-825def285843 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.721016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2bab29-6916-4456-a4e5-825def285843 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.721530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2bab29-6916-4456-a4e5-825def285843 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.727124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 278/559] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:05.728123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2bab29-6916-4456-a4e5-825def285843 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:05.771532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:05.812902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56dbfe-05e4-48dd-9e55-3ee0c2fcfcd0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.825136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:05.852840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.853850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.854343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.854879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.855781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:05.856144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2bab29-6916-4456-a4e5-825def285843 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:05.856520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:05.858101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414ca947-c1bd-4367-aee3-2b0725dd6211 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:05.871851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 282/560] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:05.873551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.874064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.874547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.898688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:05.900370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:05.909454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e5e96c-8b27-4581-b57b-41f4a292b59d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:05.910085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e5e96c-8b27-4581-b57b-41f4a292b59d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:05.910366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e5e96c-8b27-4581-b57b-41f4a292b59d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:05.910641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e5e96c-8b27-4581-b57b-41f4a292b59d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:05.978706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 283/561] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:05.980122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.024816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e5e96c-8b27-4581-b57b-41f4a292b59d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.027208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e5e96c-8b27-4581-b57b-41f4a292b59d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.072947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.091349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.094491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3fbf74-0c4c-4a76-891e-63c07e7de36b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.129805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.130465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.130750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.131052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.138481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.139958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.147676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 279/562] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:06.149147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97b92ba-5d23-47e4-916f-635a710e4b44 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.149631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97b92ba-5d23-47e4-916f-635a710e4b44 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.150097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97b92ba-5d23-47e4-916f-635a710e4b44 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:06.151547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:06.152406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:06.153349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 284/563] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:06.154346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.154719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 280/564] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:06.162539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97b92ba-5d23-47e4-916f-635a710e4b44 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.271176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de5040-3f6d-4b76-979e-0fbfb8fc29bb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.293010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97b92ba-5d23-47e4-916f-635a710e4b44 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.312724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.313461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.313808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.314145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.324818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.326321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.332669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.333277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.333560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.333833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.334521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:06.334824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.336292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.337863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f61a53ec-668c-4ef0-9fd9-32419ac1ae8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:06.342449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:06.343401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:06.344311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:06.344311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:06.344311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:06.344311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 285/565] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:06.345076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.345191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.345420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.359208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 281/566] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:06.359208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.364999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:06.366172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:06.374403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.436319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 286/567] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:06.437912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.461102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa95542f-073e-44da-84d3-6f50e45dceab None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.494510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.495220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.495503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.495778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.505710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:06.512454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:06.513353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:06.514189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 282/568] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:06.514868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.544064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0f3f1-9a9f-4401-9820-f78c1c092b3c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.571885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.581847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.582449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.582737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.583032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.583677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:06.583978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.585347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c10398-cf42-4848-9159-a3c1f0bec2d2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.603330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 287/569] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:06.603330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a73e15-8a4a-4afe-b8a0-058730a3f826 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.603330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a73e15-8a4a-4afe-b8a0-058730a3f826 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.603330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a73e15-8a4a-4afe-b8a0-058730a3f826 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.603330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 288/570] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:06.603330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a73e15-8a4a-4afe-b8a0-058730a3f826 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.606299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.627731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7060b69-b5f1-4a77-9344-239bf33f1347 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.661476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.662126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.662407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.662682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/0dd190c922964d51af263442650b427d/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.678391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=0dd190c922964d51af263442650b427d, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.679886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.695123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 283/571] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:06 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/0dd190c922964d51af263442650b427d/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:06.696655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.729681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a73e15-8a4a-4afe-b8a0-058730a3f826 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.769111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.769645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.769923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.770192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.770872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:06.771233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.772901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7bbf4c-f016-4b8f-b3b1-05a10c2a8508 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.782630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 289/572] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:06.784082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.784363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.784635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.804779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:06.806177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:06.814954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d902627-f7c1-4ad6-931e-6179e1000a96 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:06.817432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.855007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:06.855771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:06.856055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:06.856333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:06.864022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:06.865495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:06.881961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:06.882878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:06.883852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 284/573] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:06.884326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:06.889450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:06.899142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 290/574] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:06.900116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.005395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e56d40-26ac-4f51-bda9-ab3d8d23809c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.018286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1295224-6445-48ad-ba8f-582607ab3f6a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.050622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.052064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.052591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.053127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.056447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.057258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.058002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.058380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.066508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:07.069714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.071469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:07.074150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:07.075267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:07.076115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:07.083514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 285/575] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:07.084614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:07.092183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:07.093292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:07.094374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:07.094374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:07.094374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:07.095701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 291/576] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:07.096732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.117101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.157626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.202590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54578e-a062-46b3-a5ac-32ee6c7e3d2d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.219585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6002ca9-aa31-4e42-bc25-ba5730a6ef3b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.240309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.240927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.241212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.241483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.242183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:07.242482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.243965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.259389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 286/577] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:07.259389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.265725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.266820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.266820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.267043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.267578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:07.267882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.269247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.277805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 292/578] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:07.278582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.358795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.374999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9181be51-1a97-4454-bf69-a642ea218a1d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.387676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873f208-2075-4f4d-8884-24f289c8c459 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.396979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.412921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.413530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.413805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.414078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.414717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:07.415047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.416455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.423129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.423761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.424044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.424320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.433304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 287/579] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:07.433304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.440210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:07.445516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:07.446429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 293/580] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:07.447683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.541689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4b73b4-cd73-4395-991b-dcb663317667 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.556269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afbce05-511e-47aa-b3d2-7d492b05fec9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.580663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.581541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.581973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.582423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.583348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=6a835e652fcc43ab923c9c6ab434d208) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:07.583806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.585394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.602969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 288/581] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/role_assignments?user.id=6a835e652fcc43ab923c9c6ab434d208&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:07.603297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.603417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.603519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.603519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.603519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.609402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.616501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.618135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:07.637323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:07.638521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:07.639445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:07.639445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:07.639445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:07.639630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 294/582] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:07.642954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.672640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.727593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89567042-e567-43b0-bdff-e635f94d65e9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.778778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd24a2f-8db9-4230-8bc9-2bf574687d9c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.784603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.785265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.785547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.785821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.798526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.799672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:07.811412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:07.812369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:07.813342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:07.817202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 289/583] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/users/cinder => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:07.818196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.831843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.833280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.833855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.834311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.835497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:07.836467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.838307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.850390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 295/584] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:07.851655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:07.901338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.942079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e944e56-67c8-49ef-ac03-43a8601dff62 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:07.978028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:07.978227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:07.978774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:07.980209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:07.980578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:07.981261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:07.981574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:07.983490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:07.996392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 290/585] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:07.997273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.004530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11db1478-5637-43fc-bc36-afe636f030db None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.052616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.053866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.054390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.055010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.056226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:08.056929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.059893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.083663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 296/586] 10.222.0.20 () {56 vars in 1137 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:08.084760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.107048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedb556-f1ad-4a07-93bf-5b7c60ddfde0 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.143328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:08.155836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.157094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.157809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.158239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.173220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.174923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:08.196359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:08.197544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:08.198687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:08.198687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:08.198687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:08.205070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 291/587] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:08.205070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.213422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb43336-a0e5-4117-9e6d-37656c4d6a66 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.259721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.260772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.261469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.262240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.272980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.275487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.280214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:08.294721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:08.295723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 09 04:26:08.296633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:08.299360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 297/588] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/users/neutron => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:08.300840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.324192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b2812-9f10-4af0-a3c5-1b676e435219 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.360387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.361184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.361560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.361931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.374168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.375584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:08.390461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:08.391785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:08.392860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:08.392860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:08.392860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:08.398412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 292/589] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:08.399395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.432961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:08.444407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c085f13a-16cc-4b2f-b7bb-533e2810b767 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.488330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.489350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.490164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.490593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.491691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:08.492174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.494235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.514978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f6717d-dbcd-4e82-a9d8-5712e1a38d5c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.515463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 298/590] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:08.517622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.563218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.564969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.565410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.565988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.566882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:08.567385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.568980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.591777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:08.596957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 293/591] 10.222.0.20 () {56 vars in 1139 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:08.598681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.639957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f2d6cf-d42a-4859-8ddc-e882cbf7271c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.687330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.688153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.688936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.689364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.690474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:08.690949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.692937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.705115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:08.707780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 299/592] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:08.708711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.718301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128015a-3133-4dc3-84c8-4c51600a42c3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.756860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.757508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.757758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.758034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] PATH_INFO: `/v3/users/glance` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.768988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.770730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:08.786847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:08.787859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 09 04:26:08.788793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:08.791022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 294/593] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/users/glance => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:08.792262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.835148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a52af-184b-436c-934e-e610c271a9af None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.859029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:08.874877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.875592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.875870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.876171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.888804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.890257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:08.910146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:08.911309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:08.912255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:08.912255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:08.912255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:08.912255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 300/594] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:08.912503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:08.942288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def1e0f-a745-463a-bddd-d324633376cb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:08.979162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:08.980447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:08.980729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:08.981003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:08.988058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:08.989498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:09.001178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:09.002091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:09.003019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 295/595] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:09.004132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.016212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.024318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b440ff-a54a-4164-befb-0090243f1809 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.064030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.064634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.064914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.065187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.066463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:09.066784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.068401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a00177-085e-454d-92d6-055ee7e26822 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.092251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 301/596] 10.222.0.20 () {56 vars in 1137 bytes} [Fri Sep 9 04:26:08 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:09.092251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d413eb-1672-46cd-b0c3-bc3bea710330 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.092251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d413eb-1672-46cd-b0c3-bc3bea710330 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.092251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d413eb-1672-46cd-b0c3-bc3bea710330 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.092251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 302/597] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:09.092251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d413eb-1672-46cd-b0c3-bc3bea710330 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.121350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72120e00-c16b-417b-b170-2711126371c6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.125759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.156496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.157103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.157369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.157646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.158287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:09.158590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.159996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.171967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 296/598] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:09.172733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.214120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d413eb-1672-46cd-b0c3-bc3bea710330 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.246990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.256856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.257476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.257759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.258035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.258740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:09.259091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.261409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf640b94-ed87-44ae-8603-b3dda104497c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.275266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 303/599] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:09.276611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.276902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.277180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.286562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eaf089-83b0-4116-9bf9-3952de7c4ee4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.299271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:09.300484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:09.323001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.323637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.323923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.324191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.334115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.335574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:09.349997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:09.351192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:09.352100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:09.352100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:09.352100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:09.352100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 297/600] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:09.352332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.377826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 304/601] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:09.378867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.422316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.480477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ff350b-6004-421f-90a9-347f9c7f4742 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.507558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.518299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9ba072-2c83-4696-8b2e-8478e6c7fb4f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.523766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.524591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.525034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.525458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.534476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.536331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:09.551346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:09.552306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:09.553610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:09.560828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 298/602] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:09.560828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.568501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.569440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.569953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.570435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/a8add98306764181b5ea1e8a1190bbde/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.586324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=a8add98306764181b5ea1e8a1190bbde, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.588086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.609528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 305/603] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:09 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/a8add98306764181b5ea1e8a1190bbde/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:09.609528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.703644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3aa779-1e87-4666-82b2-9d1f701aae9f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.716310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.753547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.754612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.754899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.755202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.756145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:09.756458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.758176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.763511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b25435-d2a2-40bc-a85d-b83bf9acc217 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.779003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 299/604] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:09.780183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.809626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.811336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.812145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.812638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.813845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:09.814845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:09.817131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:09.833476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 306/605] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:09.834661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.846998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.910652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155104cb-c45e-47dc-97b1-b31f23f53ea4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.948795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061935a-4eab-4cd9-a2e7-4a0340bb96f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.949431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061935a-4eab-4cd9-a2e7-4a0340bb96f3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.949563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6359b-a293-491e-b039-9396c8e36956 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:09.949748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061935a-4eab-4cd9-a2e7-4a0340bb96f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.949979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061935a-4eab-4cd9-a2e7-4a0340bb96f3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.961212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1061935a-4eab-4cd9-a2e7-4a0340bb96f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-1061935a-4eab-4cd9-a2e7-4a0340bb96f3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:09.966225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:09.967197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:09.968048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 300/606] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:09.972169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efceca5-0cf4-4f1d-a51e-adb4b5275d31 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.972712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efceca5-0cf4-4f1d-a51e-adb4b5275d31 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.973539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efceca5-0cf4-4f1d-a51e-adb4b5275d31 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.974992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 301/607] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:09 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:09.980031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efceca5-0cf4-4f1d-a51e-adb4b5275d31 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:09.982833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:09.989413 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:09.990047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:09.990326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:09.990600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:09.998721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.000197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:10.011695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:10.012598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 307/608] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:10.014308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.092722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efceca5-0cf4-4f1d-a51e-adb4b5275d31 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.094278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:10.119492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eaa298-f9a1-4cd1-b110-6f24b7517451 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.131380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.132113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.132464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.132913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/4c501a6bee5b4f3a8973f727f9ef10cb/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.148017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=4c501a6bee5b4f3a8973f727f9ef10cb, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.153089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.156140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.156753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.157086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.157336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.158092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:10.158419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.159689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d7a727-686a-442e-ad05-5de5a00222b6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.168592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 308/609] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:10.169074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 302/610] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:09 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/4c501a6bee5b4f3a8973f727f9ef10cb/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:10.179237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.179237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.179237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.180033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.194106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:10.195489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:10.258428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:10.294880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 309/611] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:10.296190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.327112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0fc2fa-4108-4714-9db7-e72573c0c404 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.370588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.371602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.371859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.372136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.372841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:10.373147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.374594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.390706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 303/612] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:10.396337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.415539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:10.443817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3c31b-cca3-4656-ab43-1211149189bc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.500635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.501291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.501712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.501935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.522831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.524227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.526971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5aa15-83e2-4919-82f8-4ef931fe12ae None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:10.563954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:10.564968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:10.565841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:10.565841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:10.565841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:10.566021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 310/613] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:10.569220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.569886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.570163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.570438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/fa7372a6f3ba4f3786becc1389147d2d/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.572011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.588067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=fa7372a6f3ba4f3786becc1389147d2d, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.590162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.607165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 304/614] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:10 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/fa7372a6f3ba4f3786becc1389147d2d/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:10.638711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.659375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:10.745976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d07c25-9d7f-47e6-94f6-e874d4ec9925 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.786717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.787373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.787736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.788131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.788782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:10.789084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:10.791089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:10.808786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:10.811410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 311/615] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:10.817578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.857147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3d8b27-c0fc-4427-b989-0f23b0e74c56 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.907681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:10.909286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:10.909581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:10.909856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:10.939822 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:10.944729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:10.945653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:10.946491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:10.955362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 305/616] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:10.961466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3380a964-7b0b-4608-919c-ef07cf1c2654 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:10.962143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:10.974527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:10.999575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.000739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.001579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.002055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:11.011347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:11.013646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:11.035056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:11.036092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:11.037075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:11.044327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 312/617] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/users/nova => generated 82 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:11.049825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:11.189123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ffbb5-6211-477e-b2e0-aa5d9686fb22 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:11.196009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0c04df-c793-43c8-9229-0fd8e305afee None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:11.232456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.233179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.233480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.233731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:11.234462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:11.234774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:11.236370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:11.239111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:11.258357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 313/618] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:26:11 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:11.265434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:11.268790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.270136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.270422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.270697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:11.272698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:11.273005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:11.279526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:11.295338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 306/619] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:11.296990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:11.301836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:11.421230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c57c0af-806f-48aa-aea0-d2dd3b37beef None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:11.436312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31cedaf-19f2-4644-9181-349dff5ed58d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:11.463653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.464327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.464607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.464879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:11.489712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:11.491200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:11.506111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.506770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.507080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.507355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:11.525315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:11.526428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:11.527732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:11.527732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:11.527732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:11.527732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 307/620] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:11.533653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:11.612076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:11.612936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:11.614410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:11.629147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:11.714277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7790cc8b-b0ee-4d79-b23e-024a9fcc5e7f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:11.744785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 314/621] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:11.747895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:11.777223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.777835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.778115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.778381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:11.779050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:11.779351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:11.780724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:11.802578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 308/622] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:11.806892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:11.951248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b078618-9b89-47bc-86f8-c23693539740 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:11.966715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:11.989149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:11.990143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:11.990635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:11.991205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.001960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.003762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.012052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.020955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4628d5c1-d6b7-4939-ab55-fd20c3f61969 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:12.021671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:12.022589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:12.023992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:12.023992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:12.023992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:12.027098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 315/623] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:12.032373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.083081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.083725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.084060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.084637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.095318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.097447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ef832c60-f116-4a62-8720-bb8310f12289 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:12.112902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:12.113888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:12.115523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:12.119009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 309/624] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:12.120774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e826211-f375-4ccb-8994-d3c8c92b6c87 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.122799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e826211-f375-4ccb-8994-d3c8c92b6c87 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.123125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e826211-f375-4ccb-8994-d3c8c92b6c87 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.125891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 310/625] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:12 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:12.130838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e826211-f375-4ccb-8994-d3c8c92b6c87 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.153668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57fc08b-20ab-47b2-ab0e-a71bae7f8dd7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.189836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.190584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.191049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.191517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.192322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:12.192737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.194208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7c197b-1a51-42e5-b05c-4a007ff0783b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.212493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 316/626] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:12.212493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.212493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.212493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.229580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:12.230955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:12.250093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.285855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e826211-f375-4ccb-8994-d3c8c92b6c87 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.315127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 317/627] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:12.315127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.351028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.352051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.352550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.353016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.353987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:12.355207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.357845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.393005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 311/628] 10.222.0.20 () {56 vars in 1135 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:12.397874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.422648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.530485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2efe4e-2cd6-479e-b4a6-df112363d658 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.532856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca413389-4853-48a1-9029-c1e55bf6396f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.579728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.580710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.582269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.583338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.593916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.594848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.595353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.596340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.603646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.606195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.608019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.609565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:12.618976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:12.620527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:12.621403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:12.624082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 318/629] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:12.631178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:12.645620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:12.646970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:12.647932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:12.647932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:12.647932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:12.655482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 312/630] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:12.655482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.663761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.729925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.763844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25699b5-c422-4cdb-b698-32c07b881ff7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.772531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877accf-8205-41cd-a87e-66c31d77d008 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.799370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.800011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.800295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.800694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.801384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:12.801688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.803243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.809957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3ea9c5-2786-4aa9-8bd3-5bd265fad760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.810593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ea9c5-2786-4aa9-8bd3-5bd265fad760 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.811390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ea9c5-2786-4aa9-8bd3-5bd265fad760 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.811685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ea9c5-2786-4aa9-8bd3-5bd265fad760 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.820491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 319/631] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:12.820491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.826531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3ea9c5-2786-4aa9-8bd3-5bd265fad760 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-9f3ea9c5-2786-4aa9-8bd3-5bd265fad760 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:12.831936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:12.832835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:12.833681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 313/632] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:12.835208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c66bc4-59a7-4090-a238-cdd70bc2fd42 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.835208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c66bc4-59a7-4090-a238-cdd70bc2fd42 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.835208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c66bc4-59a7-4090-a238-cdd70bc2fd42 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.836492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 314/633] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:12.841058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c66bc4-59a7-4090-a238-cdd70bc2fd42 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.904285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.930952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640d5d7a-a63e-4203-984c-f5cafc982ca1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.948746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c66bc4-59a7-4090-a238-cdd70bc2fd42 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:12.968264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.969177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.969710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.970299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.971314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:12.971800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.973147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:12.974621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.983144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:12.983744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.984019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.984288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:12.985010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=0dd190c922964d51af263442650b427d) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:12.985314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:12.986733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3475fab8-8ea6-41f9-825e-fbf8107c67f9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:12.988526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 320/634] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:12.989512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:12.997335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 315/635] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/role_assignments?user.id=0dd190c922964d51af263442650b427d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:12.998584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:12.998870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:12.999175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.021500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:13.022751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:13.112082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 316/636] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:13.113564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.129729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b4dbd0-5cac-4804-a025-8c699f9ccaf9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.177751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.178491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.178776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.179102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.179902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:13.180209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.181626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c17c95-b660-464d-9825-d837bd3d8487 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.183731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:13.196846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 321/637] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:13.200325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd8425-ecca-46a8-a483-0e5f39bcfdac None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.200325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd8425-ecca-46a8-a483-0e5f39bcfdac None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.200570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd8425-ecca-46a8-a483-0e5f39bcfdac None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.206669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 322/638] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:13 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:13.207641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd8425-ecca-46a8-a483-0e5f39bcfdac None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.238140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189db5ef-3ac1-46fb-a601-9950f9c3a5c4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.265846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:13.287331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.287976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.288255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.288527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.298866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.300063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a65e0b75-a312-498f-9684-67938de66d71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:13.313180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:13.314179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:13.315229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:13.317207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 317/639] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:13.318557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.319013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.319434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.338244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:13.340241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:13.357010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd8425-ecca-46a8-a483-0e5f39bcfdac None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.392245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.392795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.393434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.393802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.408567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.409997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.418964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 318/640] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:13.419653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:13.427383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:13.428343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:13.429225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:13.429225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:13.429225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:13.435025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 323/641] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:13.436082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.495823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:13.532650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b677030-3d46-4402-9ff3-7aed7d75d673 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.545660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:13.568142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e835203a-85b0-489e-8943-0c943748b5ca None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.577229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.578010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.578122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.578384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.579117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:13.579427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.581149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.598695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 319/642] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:13.598695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.618544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.620437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.620886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.621600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.639841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:13.646810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:13.648050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:13.649191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:13.652577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 324/643] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:13.653612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.725393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dea2f3-f568-49c2-bfe0-8c24bd78f7a4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.763128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.763730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.764013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.764285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.764931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:13.765237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.766653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.778583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 320/644] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:13.779437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.791366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362cc50c-c385-47ae-b750-02e3cfd61b98 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.803669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:13.815421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:13.834662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.835593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.835879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.836324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/6a835e652fcc43ab923c9c6ab434d208/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.855799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=6a835e652fcc43ab923c9c6ab434d208, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.858480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.872677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 325/645] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:13 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/6a835e652fcc43ab923c9c6ab434d208/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:13.873844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:13.889473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef740da1-0d51-4953-ba30-03cd1adc95e2 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:13.936687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:13.937091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:13.937340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:13.937665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:13.938432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:13.938787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:13.940924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:13.955857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 321/646] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:13.956862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.017812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cc186a-7ff0-44c1-ae62-8c876eeb2ba0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.059730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.060730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.061209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.061675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.065186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.075161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:14.076586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab76a241-1082-4be5-a217-28ef5a875e3e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:14.080605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:14.081659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:14.082567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:14.083303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 326/647] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:14.084360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.110085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.131887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.133078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.133369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.133651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.145527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.147005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:14.163780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:14.164736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:14.165662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 322/648] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:14.171076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.204292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e32dcc2-bd11-44d9-8eeb-e1cf81a4117e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.241249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.241249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.241249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.241249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.254313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.256578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:14.279773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:14.280746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:14.281632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:14.281632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:14.281632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:14.288708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 327/649] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:14.288708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.318577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2a909d-21ad-4d28-9443-415af492dfc3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.355590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.356192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.356523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.356794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.357462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:14.357760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.359232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.368218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 323/650] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:14.369160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.390839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.392149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.425933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fc43f-99b8-4d4f-a8cc-60092945c85e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.462507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.463080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.463471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.463820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.464744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:14.465065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.466691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.486327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 328/651] 10.222.0.20 () {56 vars in 1137 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:14.487199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.518417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b587a7-70ff-40f1-b7da-e27388372f4d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.571989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.572598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.573393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.573679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.574328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:14.574891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.576203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.593579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 324/652] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:14.593579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.611614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8f990e-9f4d-492d-a23a-bbf43224f55d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.648849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.662600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.663875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.664992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.665490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.674543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.682874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.685410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:14.703542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:14.704517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:14.705476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:14.705476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:14.705476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:14.705476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 329/653] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:14.705698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.709323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955e40b6-356a-4567-969e-b13b24d18519 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.758590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.759918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.760930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.761428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.826266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05fbb62-c2c9-4f30-9a13-cdb24204cae6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:14.844649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.846139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.875884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:14.877096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:14.877962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:14.878701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:14.888913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:14.891364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:14.908190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:14.909142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 09 04:26:14.910073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:14.915467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 330/654] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/users/neutron => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:14.917792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.934268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:14.969707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 325/655] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:14.970235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:14.978510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.063331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a650a29c-75c6-4de8-9414-1ff82740dd18 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.086466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92be9c18-2e3e-4018-975c-634422064e40 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.109437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.110506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.111016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.111707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.112937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:15.113643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.115886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.124054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.124839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.125217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.125597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.126656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:15.126952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.128900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.133377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 331/656] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:15.134386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.153730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 326/657] 10.222.0.20 () {56 vars in 1139 bytes} [Fri Sep 9 04:26:14 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:15.154691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.188974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.270474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.274616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d20d6b-1535-46bc-9ab8-2b4d8c7d7d60 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.291425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b570e77-7ed5-455d-91b0-4dfbcfdfbb3b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.316551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.317448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.317903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.318327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.334917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.336291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.336897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.337128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.337507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.337913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] PATH_INFO: `/v3/users/glance` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.347038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.348344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:15.357069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:15.358200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:15.359309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:15.359309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:15.359309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:15.360699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 332/658] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:15.361619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:15.364425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:15.365534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 09 04:26:15.366663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:15.368100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 327/659] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:15.372690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.469974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.481101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabafe8-9147-4e91-91ba-9ba8fa8eeaba None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.489708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35af3905-8651-4071-8485-8c7366fea488 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.520195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.520804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.521082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.521355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.521996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:15.522301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.523762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.527541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48175426-5e2c-4b1d-96fd-03de3a876ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.528383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48175426-5e2c-4b1d-96fd-03de3a876ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.528846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48175426-5e2c-4b1d-96fd-03de3a876ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.529301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48175426-5e2c-4b1d-96fd-03de3a876ee2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.540590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 328/660] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:15.540590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.554415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.618517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48175426-5e2c-4b1d-96fd-03de3a876ee2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.620395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48175426-5e2c-4b1d-96fd-03de3a876ee2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.653948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170bf36-08ab-43e2-92d7-487c361b38a7 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.706195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.706817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.707142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.707417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.708152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:15.708457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.709897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.728645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 333/661] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:15.730041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4789b7-740d-4f6c-8d21-30c4cbdfc05a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.730328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4789b7-740d-4f6c-8d21-30c4cbdfc05a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.730597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4789b7-740d-4f6c-8d21-30c4cbdfc05a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.734979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 334/662] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:15.734979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4789b7-740d-4f6c-8d21-30c4cbdfc05a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.741978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 329/663] 10.222.0.20 () {56 vars in 1137 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:15.743161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.761249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.809963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:15.878051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4789b7-740d-4f6c-8d21-30c4cbdfc05a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.882407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13221f3-fda1-48f7-b3a8-b9c044d913e5 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:15.919981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.920875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.921318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.921588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.927159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:15.928354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.928871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.928980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.929457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.930442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.941916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:15.943542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e3bc4949-67f4-438a-8de6-b624e29a758a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:15.946294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:15.947216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:15.948158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:15.952083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 335/664] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:15.953515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:15.953643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:15.953896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:15.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:15.961577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:15.962658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:15.962658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:15.962658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:15.963670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 330/665] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:15.964463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:15.982229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:15.983622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:16.044606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.068007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 336/666] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:16.069046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.101860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839172de-dcf4-4edb-bd14-5c837a8ec3e1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.122816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.138668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece36f8-eae3-4ae1-b2c1-c812e45f0b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.139518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece36f8-eae3-4ae1-b2c1-c812e45f0b29 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.139975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece36f8-eae3-4ae1-b2c1-c812e45f0b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.140248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece36f8-eae3-4ae1-b2c1-c812e45f0b29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.188472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d340071f-2433-4dd7-a417-47b10ad0cbab None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.212961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aece36f8-eae3-4ae1-b2c1-c812e45f0b29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.214456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aece36f8-eae3-4ae1-b2c1-c812e45f0b29 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.223046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.223654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.223934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.224215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.224906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:16.225216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.226631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.246701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 337/667] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:16.246701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.283356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.326496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 331/668] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:16.328165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f172fc-e0fb-4e40-932f-848ca43fe1bc None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.328457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f172fc-e0fb-4e40-932f-848ca43fe1bc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.328733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f172fc-e0fb-4e40-932f-848ca43fe1bc None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.330405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 332/669] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:16.331450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f172fc-e0fb-4e40-932f-848ca43fe1bc None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.355789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc71c58-d4fb-43f4-8f45-359d575363e8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.357312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.389195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.389725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.390005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.390276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.390914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:16.391263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.392638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.404454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 338/670] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:16.405240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.439094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f172fc-e0fb-4e40-932f-848ca43fe1bc None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.472215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.472811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.473095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.473352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.474064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=a8add98306764181b5ea1e8a1190bbde) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:16.474368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.475839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d0aef-dbb0-4a22-a456-9c768d20e4a5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.481392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 333/671] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/role_assignments?user.id=a8add98306764181b5ea1e8a1190bbde&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:16.482577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.482861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.483161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.502577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:16.503874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:16.505105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc63b26-96bc-4b68-b12c-9f7be1a4fef3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.526900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.543550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.544197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.544476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.544753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.558246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.559788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:16.575747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:16.576802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:16.583136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:16.583136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:16.583136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:16.583136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 339/672] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:16.583136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.595575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 334/673] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:16.595575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.610510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.730872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d18d3-9a0b-4b01-913a-2699a8cccd7f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.736648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e2da3e-d877-45bb-9956-c414f9048525 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.771135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.771775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.771897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.772085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.772334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.772440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.772700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.772973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.779780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.781380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.783658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:16.789659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:16.790567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:16.791488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 340/674] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:16.792727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:16.794875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:16.795950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:16.797052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 335/675] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:16.803720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.827909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.880699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:16.914996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8a1952-a26b-41f2-a261-2320a68824b6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.923493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cdfffb-ab2a-4ada-8024-84b91ce4664d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:16.954068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.954676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.954982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.955264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.955911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:16.956219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.957592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.960186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:16.960786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:16.961062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:16.961334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:16.961991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:16.962292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:16.963699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:16.969966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 341/676] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:16.970768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:16.976602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 336/677] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:16.977576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.061010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.079158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8476fb-fd5c-4835-a9f4-4b43fcfc2b19 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.084337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faab1f5-81b8-471b-bf18-39b8b8b1be80 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.116973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.117639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.117927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.118200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.118885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:17.119249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.119906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.120532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.120804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.120900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.121074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.122548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.132214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:17.134651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 342/678] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:17.135408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:17.143246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:17.144190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:17.145025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 337/679] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:17.145713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.250597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41e2059-313d-4807-bb7b-bd30801b1a1d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.251314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3225632-c771-4697-bc36-266fa18b4651 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.286655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.287317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.287598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.287870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.289233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.289980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.290265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.290675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.295309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.296753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.299694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.304440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.306475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:17.308346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:17.309240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:17.310136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 343/680] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:17.310636 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:17.323906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:17.324944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:17.330759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:17.330759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:17.330759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:17.330759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 338/681] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:17.330759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.372228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.425112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcac9f18-4f50-4217-b442-ef1b7cecd246 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.435683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6672eab0-343e-4202-b77e-5da5511785ba None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.461148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.461770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.462051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.462321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.463097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=4c501a6bee5b4f3a8973f727f9ef10cb) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:17.463438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.464980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.471485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 344/682] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/role_assignments?user.id=4c501a6bee5b4f3a8973f727f9ef10cb&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:17.472100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.488363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.488996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.489275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.489557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.490244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:17.490552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.491985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.508433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 339/683] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:17.509397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.582552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.595519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62126199-8f38-4f3e-8fd0-4744bb8eea44 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.620653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.634072 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.634789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.635804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.636219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.637127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:17.637541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.639137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.647105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001e3182-8fe7-46c3-a2e3-4311c68db662 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.654623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 345/684] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:17.655815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.699660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.700362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.700732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.701071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.701792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:17.702204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.704963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.720884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 340/685] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:17.721745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.785820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6901c8af-7183-4836-a302-7a9b5bc8b497 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.826843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.828045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.828330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.828758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.840602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093ef1ef-2d86-492e-916c-c1f01a5f3ffb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:17.842600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.845028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.858172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:17.865980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:17.867001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:17.868533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:17.868533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:17.868533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:17.872489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 346/686] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:17.873361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.887795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:17.888843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:17.889336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:17.889758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:17.890687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=fa7372a6f3ba4f3786becc1389147d2d) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:17.891212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:17.893053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:17.904215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 341/687] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/role_assignments?user.id=fa7372a6f3ba4f3786becc1389147d2d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:17.904215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:17.915739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.074416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b75964-c979-4ac8-b367-a1895fdc3ace None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.092253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba933353-5bac-484d-95ed-149e9bd5fd01 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.132375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.133726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.133987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.134351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.134628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.134904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.135046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.135671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.136611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:18.137113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.139655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.144507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.146018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.155225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 342/688] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:18.160190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:18.161132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:18.162676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 347/689] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:26:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:18.164341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:18.167940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:18.211241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.282170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.293752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1d80f8-f543-43d3-8567-acfbcc279f9b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.296665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b201d-9c48-4fdc-99c7-95b54c8ddd3a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.337385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.338119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.339071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.339356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.340856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.341567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.341850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.342220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.343026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:18.343283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.344719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.347131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.349366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:18.363739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:18.364820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:18.365741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 343/690] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:18.367277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 348/691] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:26:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:18.370747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:18.372379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:18.487314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74c288-2c32-4aff-b823-bf8ae06047d2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.489066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.499585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba98d344-feaf-4cd0-b8fa-1f553d31591c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.530270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.530957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.531269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.531541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.545142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.545939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.546664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.547086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.548276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:18.548694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.550495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.560007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.578970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 344/692] 10.222.0.20 () {56 vars in 1135 bytes} [Fri Sep 9 04:26:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:18.582086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:18.629441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.631482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.708303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae71e41-c740-4fac-a87f-c3e3682aef90 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:18.755626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 349/693] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 09 04:26:18.762772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:18.770870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01409975-b417-471d-8546-ee93e2657b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:18.771820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01409975-b417-471d-8546-ee93e2657b9d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:18.772384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01409975-b417-471d-8546-ee93e2657b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:18.772852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01409975-b417-471d-8546-ee93e2657b9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:18.816788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.867823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01409975-b417-471d-8546-ee93e2657b9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:18.869195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:18.871182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01409975-b417-471d-8546-ee93e2657b9d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:18.968727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cfd2b-e40b-43ae-abeb-aca22f0834c8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.015499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.016605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.017138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.017625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.023204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 345/694] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 09 04:26:19.024556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a651141f-dd83-45bb-bcf0-c2a7cdeb3ed0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.024846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a651141f-dd83-45bb-bcf0-c2a7cdeb3ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.025121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a651141f-dd83-45bb-bcf0-c2a7cdeb3ed0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.026969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 346/695] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:19.028328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a651141f-dd83-45bb-bcf0-c2a7cdeb3ed0 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.035295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.036836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-42dd4f4d-9809-41b2-ba05-3effe3538090 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:19.055223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:19.056207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:19.057158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:19.057158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:19.057158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:19.057158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 350/696] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:19.058886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.059420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.059791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.080273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:19.081642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:19.092758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.142493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a651141f-dd83-45bb-bcf0-c2a7cdeb3ed0 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.171013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 351/697] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:19.172357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.199264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.200236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.200663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.201142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.220758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.223783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.241267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:19.266802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:19.267967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:19.269068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:19.269068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:19.269068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:19.271657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 347/698] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:19.276545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.382548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024d2d71-da9a-4dcf-ab41-58095f0bb4a3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.436578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8095b79-7f65-4025-9632-c1d34de446ec None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.453077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.464705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.465326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.465603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.465875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.466588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:19.466896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.468285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.476475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.477294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.477720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.478139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.479011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:19.479984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.481553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.495677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 348/699] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:19.496763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.497057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 352/700] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:19.501470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.607448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebe0869-570f-4196-990b-f48e61d1b4fa None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.611075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.622317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c49974-a34c-4c7b-a5b9-f489c426fc11 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.646498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.647404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.647896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.648342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.649173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.651356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.661577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.662337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.662743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.663142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:19.664488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:19.665715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:19.666745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:19.669405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 349/701] 10.222.0.20 () {56 vars in 1129 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:19.670526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.678208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.679807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:19.694630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:19.695659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:19.696537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:19.699002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 353/702] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:19.699975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.711666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.803301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9125b39-372e-460d-b74e-a073af8c171c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.818841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fde0af-5395-4217-b9ed-e3eb59513e5a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:19.846358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.847365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.847723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.848069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.857129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.858733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.859011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:19.859835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:19.860295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:19.860754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:19.861560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:19.862070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:19.863679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:19.868792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 354/703] 10.222.0.20 () {56 vars in 1124 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:19.870233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:19.874986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:19.875912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:19.876845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:19.879342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 350/704] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:19.880376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:19.893892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.983896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:19.995178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a0a801-8485-49c6-9e48-572904d05ec5 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:20.001348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7601c7a-5cf7-47ca-9667-44a5f1fdbfa2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:20.032820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:20.033881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.034342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.034776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.035446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:20.035753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:20.037422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:20.038797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:20.039437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.039719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.039993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.040684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:20.040989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:20.042433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5772a441-f4b8-477a-b7dd-30b65cbe8514 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:20.044517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 351/705] 10.222.0.20 () {56 vars in 1124 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:20.045536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:20.064349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 355/706] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:20.066241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab66f4-04a9-4c1e-a069-b059ac8a7182 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.066751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab66f4-04a9-4c1e-a069-b059ac8a7182 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.067237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab66f4-04a9-4c1e-a069-b059ac8a7182 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.069431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 356/707] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:20.070519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ab66f4-04a9-4c1e-a069-b059ac8a7182 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:20.167195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:20.169382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e6bf4d-4085-4701-8b78-8bd60eb1bee6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:20.198367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ab66f4-04a9-4c1e-a069-b059ac8a7182 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:20.215862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:20.217179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.217466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.217742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.218466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:20.219066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:20.220915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e617d2ad-9ce0-454f-b1a9-f03784e1c51a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:20.244008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 352/708] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:20.245410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.245695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.245970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.248118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:20.249082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.249585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.250147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.251183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=6a835e652fcc43ab923c9c6ab434d208) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:20.251767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:20.253455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:20.271044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:20.271233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 357/709] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:20 2022] GET /identity/v3/role_assignments?user.id=6a835e652fcc43ab923c9c6ab434d208&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:20.271233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:20.271374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:20.280738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:20.374170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 353/710] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:20.431621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a5874-0a23-4ca6-9409-fc40f85a8710 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:20.470943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:20.471578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.471855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.472126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.472853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=0dd190c922964d51af263442650b427d) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:20.473156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:20.474640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d095da52-8830-40e4-90e8-4c5c0e53d4a7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:20.481515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 358/711] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:20 2022] GET /identity/v3/role_assignments?user.id=0dd190c922964d51af263442650b427d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:20.701535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:20.706875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ecfd32-da00-439d-8edc-4520a43e54d5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.707200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ecfd32-da00-439d-8edc-4520a43e54d5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.707477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ecfd32-da00-439d-8edc-4520a43e54d5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.709064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 359/712] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:20.715216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.715506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.715766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.724103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:20.744357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:20.745628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:20.811232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003bbbf-fd85-4c70-8371-052089f9b2d4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:20.826162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 360/713] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 09 04:26:20.848207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:20.848844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:20.849122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:20.849391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] PATH_INFO: `/v3/services/network` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:20.849988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:20.851407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:20.861952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:20.863134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:20.863975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 354/714] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:20 2022] GET /identity/v3/services/network => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:20.871733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:20.999405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ae9ca-37d8-4f88-929d-68261db3e3ca None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:21.030213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:21.040734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:21.041370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:21.041685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:21.041956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:21.042588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] RBAC: Adding query filter params (name=network) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:21.042895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:21.044276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:21.049158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 361/715] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:20 2022] GET /identity/v3/services?name=network => generated 124 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:21.054731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:21.070739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:21.191888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3814f8-e613-4471-9857-994f1e43c525 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:21.227755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:21.228708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:21.229246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:21.229711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:21.230583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] RBAC: Adding query filter params (type=network) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:21.231113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:21.232704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:21.237881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 355/716] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:21 2022] GET /identity/v3/services?type=network => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:21.292745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:21.436091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371275b8-40a6-4608-b0c0-1961b135d012 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:21.474008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:21.475167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:21.475727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:21.476258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:21.490062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:21.491935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:21.509023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:21.510065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:21.510963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:21.510963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:21.510963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:21.513096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 362/717] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:21.519121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:21.620447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:21.655531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9402215d-2d10-4c3a-b5a5-60269010fcf3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:21.695089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:21.696063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:21.696583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:21.697077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:21.698090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:21.698586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:21.700170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640dc180-5e42-48a8-939d-b94a0970f089 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:21.712416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 356/718] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:21.719519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:21.800930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:21.868291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8286c3fe-7857-4870-94b7-2fdcd5a6a3fd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:21.921552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841bec3a-d5e7-432c-8fdb-27246674da3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:21.922168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841bec3a-d5e7-432c-8fdb-27246674da3c None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:21.922452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841bec3a-d5e7-432c-8fdb-27246674da3c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:21.922729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841bec3a-d5e7-432c-8fdb-27246674da3c None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:21.923414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841bec3a-d5e7-432c-8fdb-27246674da3c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:21.924836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841bec3a-d5e7-432c-8fdb-27246674da3c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:21.961258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 363/719] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:26:21 2022] POST /identity/v3/users => generated 254 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:26:22.583310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ba990-8640-467a-9dda-a3bd16f10e05 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:22.583814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ba990-8640-467a-9dda-a3bd16f10e05 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:22.583814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ba990-8640-467a-9dda-a3bd16f10e05 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:22.585253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 357/720] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:22.590864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dc6438-edaf-425b-9e84-4524d50bf354 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:22.591349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dc6438-edaf-425b-9e84-4524d50bf354 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:22.591448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dc6438-edaf-425b-9e84-4524d50bf354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:22.614027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-13dc6438-edaf-425b-9e84-4524d50bf354 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:22.615237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-13dc6438-edaf-425b-9e84-4524d50bf354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:22.690204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 364/721] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:23.009733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a6be7-88a9-434f-96d3-64710c7d2d12 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.010649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a6be7-88a9-434f-96d3-64710c7d2d12 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.010876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a6be7-88a9-434f-96d3-64710c7d2d12 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.013026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 358/722] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:23.018323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f246fd5-bf65-4e09-ace7-8c231aa4f1ed None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.018838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f246fd5-bf65-4e09-ace7-8c231aa4f1ed None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.019391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f246fd5-bf65-4e09-ace7-8c231aa4f1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.047957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3f246fd5-bf65-4e09-ace7-8c231aa4f1ed None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:23.049514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3f246fd5-bf65-4e09-ace7-8c231aa4f1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:23.082041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:23.136950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 365/723] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:23.171898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a6be7-88a9-434f-96d3-64710c7d2d12 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:23.248500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a8875-d1d5-44b2-a291-19d6540fd319 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.249155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a8875-d1d5-44b2-a291-19d6540fd319 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.249635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a8875-d1d5-44b2-a291-19d6540fd319 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.252112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 366/724] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:23.260283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.260553 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.260773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.291666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:23.292862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:23.346795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a6be7-88a9-434f-96d3-64710c7d2d12 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:23.391006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:23.397973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:23.398900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.399508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.401258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.403018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:23.405705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:23.418528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 367/725] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:23.439170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 359/726] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:26:23 2022] POST /identity/v3/services => generated 251 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:23.479337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:23.613330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:23.621922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e39a86-6789-4ad4-b41b-826ae2a91f1c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:23.675327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:23.676347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.676835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.677331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.690267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:23.695749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:23.696767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:23.697770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:23.700593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 368/727] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:23.706083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:23.783718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:23.863214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a6001-48ec-4eb1-bfd5-c71b42f371a3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:23.906424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:23.907713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.908205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.908907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.910218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:23.911025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:23.912937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:23.920470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dc9ef-f755-415e-8e3c-8a9dfee805d3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:23.925288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 360/728] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:23.930539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:23.962476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:23.964111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:23.964396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:23.964670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:23.978982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:23.984999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:23.985922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:23.986757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 369/729] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:23.992211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.002400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.009180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.072885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaceff6-4282-44b7-b657-db51f942bc8b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.112020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.112663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.113109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.113387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.126340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.127867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-e8ef0858-baf1-437e-89f7-9c89de438c5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:24.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:24.148185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:24.150255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:24.150255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:24.150255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:24.154789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 361/730] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:24.158811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af2d06-c1fb-4070-afdf-071f4b2a90a9 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.159308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af2d06-c1fb-4070-afdf-071f4b2a90a9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.159737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af2d06-c1fb-4070-afdf-071f4b2a90a9 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.161601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 362/731] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:24.162535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87af2d06-c1fb-4070-afdf-071f4b2a90a9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.199459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08595298-226f-45fb-ab72-cb3ce67d26a8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.247095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.248379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.248815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.249485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.250577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:24.251075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.253355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626e29e0-6d87-41ee-abea-c67e1718b5fd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.272528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 370/732] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:24.273843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.274132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.274408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.294178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:24.295426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:24.312678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87af2d06-c1fb-4070-afdf-071f4b2a90a9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.323969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.352905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.353519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.354342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.354641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.355298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:24.355601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.356998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.370213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 363/733] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:24.371087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.382502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 371/734] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:24 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 09 04:26:24.384167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.388397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.493934 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce721dd3-8ad2-49cd-8838-39f5a5206d52 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.509659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8e5736-4c64-4013-a90f-507848430f81 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.545578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.546212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.546486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.546756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.554808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.555471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.555745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.556015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.568459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.570793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.572127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.573690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:24.600836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:24.601859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:24.603390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:24.603390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:24.603390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:24.603390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 364/735] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:24.603670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:24.606008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:24.610955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 372/736] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:26:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:24.612335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.616190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.638180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.742340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.778977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463bc74d-f096-4d33-b0ba-c5d38a850196 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.794679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ceb39f-8851-413a-8041-d00185bc2ce0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.819496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.820108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.820390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.820666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.821315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:24.821624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.823688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.833530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.834192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.834473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.834775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.835755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:24.835859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 365/737] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:24.835936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.837387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8773bc5-a87b-430a-9de1-c8ae64c877e8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:24.840792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.857327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 373/738] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:26:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:24.859032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289dfcd3-6386-4ea1-b6a0-570480c6e3fb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.859475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289dfcd3-6386-4ea1-b6a0-570480c6e3fb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.859905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289dfcd3-6386-4ea1-b6a0-570480c6e3fb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.861798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 374/739] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:24.862858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289dfcd3-6386-4ea1-b6a0-570480c6e3fb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:24.929422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.953670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b2ab4-c179-4f63-840c-3bdd35cb0f15 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.968612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289dfcd3-6386-4ea1-b6a0-570480c6e3fb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:24.988512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:24.991047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8616891-4a68-4080-af12-882780001210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:24.992290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8616891-4a68-4080-af12-882780001210 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:24.992678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8616891-4a68-4080-af12-882780001210 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:24.992960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8616891-4a68-4080-af12-882780001210 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:24.993601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8616891-4a68-4080-af12-882780001210 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:24.995163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8616891-4a68-4080-af12-882780001210 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.003420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.004184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.004468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.004744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] PATH_INFO: `/v3/users/swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.012550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.014070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.015087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 366/740] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:26:24 2022] POST /identity/v3/services => generated 240 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:25.015952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8616891-4a68-4080-af12-882780001210 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5242977f-826f-4127-bb67-c9ea5c0b21af None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:25.033068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:25.034110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:25.035098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 375/741] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:25.036189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.036189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.036441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.057736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:25.059132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:25.149129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 376/742] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:25.150045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.161435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8616891-4a68-4080-af12-882780001210 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.200755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f915694-84d3-44ba-8366-063998515899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.201748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f915694-84d3-44ba-8366-063998515899 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.202365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f915694-84d3-44ba-8366-063998515899 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.202839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f915694-84d3-44ba-8366-063998515899 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.215396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:25.223800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f915694-84d3-44ba-8366-063998515899 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.225477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f915694-84d3-44ba-8366-063998515899 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.262059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7874765c-edc8-476f-871d-83defb5a6bdb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-4f915694-84d3-44ba-8366-063998515899 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:25.264258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:25.265426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:25.266524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:25.266524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:25.266524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:25.268331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 367/743] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:25.271485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:25.273544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f915694-84d3-44ba-8366-063998515899 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.304148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.304696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.304950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.305217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.305943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:25.306258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.308806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.327860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 377/744] 10.222.0.20 () {56 vars in 1135 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:25.334820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.409634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f915694-84d3-44ba-8366-063998515899 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.448712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7721940-8ff5-4220-b7fd-38ba97494c39 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.459804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.460043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.460486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.460856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.461650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:25.462055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.464696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.481827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 368/745] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:25.486032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.498642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.499569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.500081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.500557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.537886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:25.550518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:25.582654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.584561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.615912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592186b-3827-4370-94fe-89ff7b2a59ae None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.662849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.663747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.664169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.664581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.674134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.676563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:25.692884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:25.693974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:25.695187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:25.697216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 369/747] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:25.698691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.702129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 378/747] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:26:25.703393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.769575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:25.811996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6aea57-574b-46f5-b132-04d8a5e8c29d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.831500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda9d17-6da2-425e-a3d2-d55158f45922 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:25.845864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:25.851799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.852638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.853179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.854119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.855171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:25.855625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:25.857203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:25.871226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 379/748] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:25.872245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.875288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:25.876967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:25.877479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:25.877944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:25.891159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:25.897949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:25.898903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:25.899756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:25.901147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 370/749] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:25.901331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:25.989944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19cca7b-9225-472d-b27f-a7dfc78a74dd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.015402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:26.027884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.028570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.028854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.029127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.034649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d6a0d-2226-44d7-a8a7-3246461dc3b1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.040523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.042088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:26.059907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:26.060902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:26.061788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:26.061788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:26.061788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:26.065821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 380/750] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:26.066649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.080955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.081991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.081991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.082164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/0dd190c922964d51af263442650b427d/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.084319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=0dd190c922964d51af263442650b427d, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.086085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.103340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 371/751] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:25 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/0dd190c922964d51af263442650b427d/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:26.104444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.143133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:26.251927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8ea62d-847a-4d48-b168-5cb61129324e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.282571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09af4a5-d3a7-4272-b23e-f4747ff272c9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.300334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.300945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.301233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.301514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.302184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:26.302492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.303925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a232b89c-a011-4fb2-bcb0-ee8ac8f313ff None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.313332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 381/752] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:26.314603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056216f1-bcd2-48c3-91a6-07418b6a7c68 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.314969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056216f1-bcd2-48c3-91a6-07418b6a7c68 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.315249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056216f1-bcd2-48c3-91a6-07418b6a7c68 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.316752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 382/753] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:26.321468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056216f1-bcd2-48c3-91a6-07418b6a7c68 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.329094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.329710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.329989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.330261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.330910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:26.331303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.332678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ae12a-d895-4b42-923e-3bce759873ca None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.350735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 372/754] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:26.352310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.352601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.352879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.372060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:26.373038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:26.374605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:26.445456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 373/755] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:26 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:26.446729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.448900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056216f1-bcd2-48c3-91a6-07418b6a7c68 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.479097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:26.494962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.495641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.495920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.496194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.508897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.514969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:26.527495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:26.528472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:26.529765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:26.529765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:26.529765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:26.529765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 383/756] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:26.535473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.648646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:26.666796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2ff3f-f3aa-4fb3-b1f6-f5618a3b3f5d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.675547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c72afe-2dce-489f-91e0-c36fd0a3d685 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.728505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.729725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.729932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.730204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.730850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:26.731171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.732522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.738496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.739153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.739425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.739691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.751729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.753170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.760367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 384/757] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:26.768300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:26.781872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:26.782781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:26.783736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 374/758] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:26.789124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.886771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afbd32b-3ca2-4293-a71c-44154113b1c1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.888995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:26.913981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0cb60-ec1e-4fd1-91cf-087f7023dbad None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:26.929906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.930907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.931311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.932215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.941657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.943445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.951350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:26.952149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:26.952576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:26.952997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:26.953856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:26.954322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:26.955947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:26.959142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:26.960155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:26.961138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:26.965854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 385/759] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:26.966748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.978804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 375/760] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:26.980094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:26.982547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.090119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc7d0db-dfff-4406-b81b-b3e4f3ba7417 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.091501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3039b689-b0c7-4a93-9294-0b1784f11e28 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.128992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.129613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.129894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.130169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.130909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:27.131265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.132254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.132799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.132924 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.133150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.133422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.134008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.135693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.143551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:27.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:27.153337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:27.154163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:27.156461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 386/761] 10.222.0.20 () {56 vars in 1129 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:27.157275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.163906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 376/762] 10.222.0.20 () {56 vars in 1149 bytes} [Fri Sep 9 04:26:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:27.163906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.255797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.295081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b103fcee-a179-473d-8be7-dea3f49ea931 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.297066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0382fba9-e3d5-4108-9c5e-099f61c74646 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.332814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.333175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.333456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.333734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.334421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:27.334808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.336163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.339832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.340685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.341076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 377/763] 10.222.0.20 () {56 vars in 1124 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:27.341372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.341878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.342102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.343151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=6a835e652fcc43ab923c9c6ab434d208) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:27.343654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.345716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.352144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 387/764] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/role_assignments?user.id=6a835e652fcc43ab923c9c6ab434d208&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:27.353256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.410041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.487235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50781542-c42b-4b71-80fb-25faad5e020c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.488679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d130c421-1b76-4013-a9f6-41cd59cbabe8 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.529826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.530786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.531265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.531734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.532557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:27.533033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.535434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.543469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 388/765] 10.222.0.20 () {56 vars in 1124 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:27.549797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.556206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.570539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b92c0a-a952-4ee9-9491-6869d17c8d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.571707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b92c0a-a952-4ee9-9491-6869d17c8d37 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.572136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b92c0a-a952-4ee9-9491-6869d17c8d37 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.573148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b92c0a-a952-4ee9-9491-6869d17c8d37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.663340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b92c0a-a952-4ee9-9491-6869d17c8d37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.665497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b92c0a-a952-4ee9-9491-6869d17c8d37 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.677350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b3ffb9-7bd7-48d7-b523-854ad52d39d4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.715167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.715867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.716215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.716552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.717319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] RBAC: Adding query filter params (interface=public, service_id=b66c1c30d31b45d5bbc23e638b533632, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:27.717697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.719209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05863785-33fa-4fc6-a71d-c9029d06edd1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.727934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 389/766] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/endpoints?service_id=b66c1c30d31b45d5bbc23e638b533632&interface=public®ion_id=RegionOne => generated 194 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:27.729386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a405b075-0c59-498f-8184-b769a39a2ea1 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.729942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a405b075-0c59-498f-8184-b769a39a2ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.730054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a405b075-0c59-498f-8184-b769a39a2ea1 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.735015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 390/767] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:27.735963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a405b075-0c59-498f-8184-b769a39a2ea1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.743872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.793701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 378/768] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:26:27.795320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.795805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.796296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.811330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:27.818527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:27.820051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:27.852935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a405b075-0c59-498f-8184-b769a39a2ea1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:27.896317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:27.897353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:27.897876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:27.903057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:27.903229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 379/769] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:27 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:27.903229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:27.904088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:27.905582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:27.920053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 391/770] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/services => generated 772 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:28.046239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:28.057064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705db96f-a1b3-4fe8-95a1-17bd6bba3215 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:28.103154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:28.104322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.104599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.104866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.121738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:28.123246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:28.147415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:28.148578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:28.149690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:28.149690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:28.149690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:28.149690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 380/771] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:28.155547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.219358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.347621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3e82ae-4782-40b7-ba91-d556a6ab5386 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:28.386566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:28.387576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.387837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.388105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] PATH_INFO: `/v3/services/network` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.388809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:28.390195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:28.400905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:28.401913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:28.402746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 381/772] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/services/network => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:28.409271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.410214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb1f686-6189-4b03-a3b9-0d37e7134054 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:28.424079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:28.446581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:28.455522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:28.456128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.456403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.456676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.457420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:28.457820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:28.459212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:28.473881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 392/773] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:28.479127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.565401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e26158c-3ca1-486a-93e1-5f47d0804d0f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:28.629241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:28.630181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.630683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.631167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.632004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] RBAC: Adding query filter params (name=network) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:28.632314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:28.633678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77efb55d-2210-4a92-abac-dbd3dbb8626d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:28.645924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 382/774] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/services?name=network => generated 124 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:28.645924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e924ff1-0d53-4774-99c1-638c78ce29a6 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.645924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e924ff1-0d53-4774-99c1-638c78ce29a6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.645924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e924ff1-0d53-4774-99c1-638c78ce29a6 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.645924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 383/775] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:28.645924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e924ff1-0d53-4774-99c1-638c78ce29a6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.662896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c72df-f0eb-4552-9486-fe4057697056 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:28.705084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:28.706390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.706880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.707397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.715540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:28.716977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:28.731309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ddd5e2fc-ad16-4d88-8bb8-6a3f4ce2b0dc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:28.732059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:28.734037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:28.735097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:28.737057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 393/776] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:28.738556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.739079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.739549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.759841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:28.761528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:28.764874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e924ff1-0d53-4774-99c1-638c78ce29a6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:28.801743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:28.802349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:28.802622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:28.802891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:28.803565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] RBAC: Adding query filter params (type=network) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:28.803863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:28.805223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:28.816254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:28.816411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 384/777] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/services?type=network => generated 350 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:28.817904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.840115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 394/778] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:28.841339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:28.990081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d47ad9-8c5e-4165-b54a-83f65d71eb0c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.018851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8848df3-76e1-46fa-9ff4-3a5275543d05 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.031213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.032029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.032405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.032775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.033576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] RBAC: Adding query filter params (interface=public, service_id=3eb8e4b86d724f6fbad74d26dba9a927, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:29.033983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:29.037967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:29.042827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 395/779] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/endpoints?service_id=3eb8e4b86d724f6fbad74d26dba9a927&interface=public®ion_id=RegionOne => generated 194 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:29.048095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:29.100631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:29.106708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.107371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.107654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.107928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.108629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:29.108941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:29.110373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:29.113952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:29.127826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 385/780] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:29.133552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:29.178437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc47b1-6c50-449b-88cd-bbf01b5bde73 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.220666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.221444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.221727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.222004 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.222651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:29.224135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:29.231726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 396/781] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:29 2022] GET /identity/v3/services => generated 772 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:29.236598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:29.266276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbb87d2-5a9f-40d0-b67d-c96342f5ef25 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.304946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.305573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.305855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.306198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.307914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=719e5e2221e04ce0aa7f303b2ea84b89) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:29.308325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:29.309783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:29.316621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 386/782] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:29 2022] GET /identity/v3/role_assignments?user.id=719e5e2221e04ce0aa7f303b2ea84b89&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:29.362226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50b63ef-efe1-49dc-8614-515d0b1e87bf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.378457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:29.412044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.412701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.412983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.413260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.425940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:29.431088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:29.433891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:29.435432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 397/783] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:29.444147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:29.538400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:29.593480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6caec1-5de7-43e1-a3c3-6f1119a96cb6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.628461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.629076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.629354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.629624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.630288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:29.630589 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:29.632006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:29.641400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 387/784] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:29.646531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:29.777809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb8ad87-3b8b-4dee-9a38-f00fafb3f4e8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:29.818944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:29.819508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:29.819848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:29.820123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:29.835281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:29.836721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:29.857177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:29.858226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:29.859189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:29.859189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:29.859189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:29.859189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 398/785] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:29.864111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:30.004661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81746cb-e687-442a-95dd-8b5dfd2bdaaf None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:30.032931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:30.051207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:30.051872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.052156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.052491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.053149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:30.053455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:30.054834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060d18af-6186-4fcd-9762-8883b8ea2679 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:30.072346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 388/786] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:30.077926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:30.084973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a9af2-2449-44a6-b188-753db01d2dcf None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.085549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a9af2-2449-44a6-b188-753db01d2dcf None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.086034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a9af2-2449-44a6-b188-753db01d2dcf None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.088545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 389/787] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:30.093790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.094328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.094819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.112915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:30.114240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:30.115567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:30.227537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 390/788] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:30.247854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26929e70-aab2-4925-924c-567971c20ba5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:30.287257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:30.287886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.288166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.288446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.296923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:30.298345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:30.310637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:30.312496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:30.314399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 399/789] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:26:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:30.318748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:30.528531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def6accf-0e57-4f13-81ee-8636be54b820 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:30.552718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:30.590279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:30.593174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:30.594198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.594688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.602983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.602983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:30.602983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:30.602983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:30.632649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 391/790] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:26:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:30.639652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:30.780877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:30.819660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddc6b46-a0d6-4481-bff2-c132cdb5431a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:30.892341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90151816-f58e-41b0-8d75-54ce21e6e260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:30.893486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90151816-f58e-41b0-8d75-54ce21e6e260 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.895069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90151816-f58e-41b0-8d75-54ce21e6e260 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.895550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90151816-f58e-41b0-8d75-54ce21e6e260 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.901517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d8161-915e-41b3-aca1-cbbe0e3b1b99 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:30.918966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90151816-f58e-41b0-8d75-54ce21e6e260 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-90151816-f58e-41b0-8d75-54ce21e6e260 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:30.924480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:30.925407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:30.926224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:30.930142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 400/791] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:30.933912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fef41-9c54-4500-9712-f5b6d4adb977 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.934193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fef41-9c54-4500-9712-f5b6d4adb977 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.934465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fef41-9c54-4500-9712-f5b6d4adb977 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:30.936256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 401/792] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:30.936947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17fef41-9c54-4500-9712-f5b6d4adb977 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:30.966026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:30.967076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:30.967609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:30.968084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.016746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:31.049891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17fef41-9c54-4500-9712-f5b6d4adb977 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.063398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.066319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.086249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.086964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.087173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.087530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.088095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:31.088395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.089869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837ab22d-8899-48bb-82aa-4d7ccf48f7f6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.102652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 402/793] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:31.104440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.104719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.104990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.111710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:31.125656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:31.126827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:31.203362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 403/794] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:31.203362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.217536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 392/795] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:30 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 579 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 09 04:26:31.225192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.351867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:31.379823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9f585b-3c97-48c7-adfa-6cb6b5cf88de None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.417446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:31.417804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69179f2-d0f2-4e02-a410-5099119efc6a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.429544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.430180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.430457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.430725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.450118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.451588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:31.479996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:31.481012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:31.482601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:31.482601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:31.482601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:31.482601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 404/796] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:31.486365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.487035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.487317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.487643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.487812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.506956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.506956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:31.535939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:31.536940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:31.537824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:31.537824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:31.537824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:31.541773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 393/797] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:31.544255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.638895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a77c5e-043d-4aaa-9ccd-fc89e3281415 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.680029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9717f0f-35da-4184-a83c-b512ccc0fb30 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.680714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.681878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.683559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.684050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.684905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:31.685420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.687100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.703010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 405/798] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:31.704249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.714398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:31.729439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.730398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.730955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.731575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.732431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:31.732933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.734742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.752814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 394/799] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:31.754003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.755448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:31.819549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb7924-18cf-4069-a09d-5850d10dbe48 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.860673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.861338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.862001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.862279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.863058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.864558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.872313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a1a6ea-be2d-4326-9a1d-cf60e41b62d9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:31.875729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:31.876891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 09 04:26:31.878118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:31.881097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 406/800] 10.222.0.20 () {56 vars in 1129 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:31.881976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.915211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:31.916857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:31.917407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:31.917900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:31.926233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:31.928269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:31.941606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:31.942565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:31.943554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:31.946356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 395/801] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:31.948064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:31.995000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c799e6bb-64a0-4e9d-a0a9-afbd8bbe0674 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:31.995228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.004784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.032058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.032710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.032983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.033259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.042484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.044010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-fb8dc0a5-e735-4529-8ead-0784f8d724ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:32.057566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:32.058613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:32.059579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:32.062647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da44694-05cc-4412-a537-3abeca82bd7a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.063852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 407/802] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:32.065057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a7c0f-43f9-41f7-8731-7d4d990337dd None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.065354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a7c0f-43f9-41f7-8731-7d4d990337dd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.065809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a7c0f-43f9-41f7-8731-7d4d990337dd None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.067418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 408/803] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:32.068129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4a7c0f-43f9-41f7-8731-7d4d990337dd None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.100437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.101120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.101371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.101567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.102184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:32.102483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.103910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.108703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 396/804] 10.222.0.20 () {56 vars in 1124 bytes} [Fri Sep 9 04:26:31 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:32.109237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.186876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4a7c0f-43f9-41f7-8731-7d4d990337dd None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.210064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eef929-4b73-4bda-bd4c-0bfba5c83c24 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.216797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.220330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.220937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.221220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.221497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.222225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:32.222531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.223997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782779ac-5527-442d-85f2-8c2f2542c750 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.241812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 409/805] 10.222.0.20 () {56 vars in 1135 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:32.243113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164621a-b448-4c83-b890-8d1eb485e8d9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.243400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164621a-b448-4c83-b890-8d1eb485e8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.243672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164621a-b448-4c83-b890-8d1eb485e8d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.249440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.250047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.250328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.250603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.251344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:32.251654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.253086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.262955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0164621a-b448-4c83-b890-8d1eb485e8d9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:32.264168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0164621a-b448-4c83-b890-8d1eb485e8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:32.265210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 397/806] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:32.266129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.290731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.337754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 410/807] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:32.339007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1010f-ebfa-4122-a0cb-18f340424bd5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.339301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1010f-ebfa-4122-a0cb-18f340424bd5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.339567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1010f-ebfa-4122-a0cb-18f340424bd5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.340942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 411/808] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:32.341469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e1010f-ebfa-4122-a0cb-18f340424bd5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.374068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0eaa6b1-65e4-4390-9427-52c3c8608a82 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.409214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.409842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.410154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.410391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.411048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:32.411364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.412721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.418085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 398/809] 10.222.0.20 () {56 vars in 1124 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:32.418085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.454565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.456038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e1010f-ebfa-4122-a0cb-18f340424bd5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.492255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.492958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.493302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.493670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.504085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.505593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8d8bbb23-7dc1-426a-9ad5-80dd863492ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:32.521822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:32.522772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:32.523789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:32.523789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:32.523789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:32.523789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 412/810] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:32.524117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.524845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.525213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.532897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9ddf4c-6f7c-4091-bb16-b9fef82ad052 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.543593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:32.544757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:32.547014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.571025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.571781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.573100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.573402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.574120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=0dd190c922964d51af263442650b427d) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:32.574436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.575890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.590805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 399/811] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/role_assignments?user.id=0dd190c922964d51af263442650b427d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:32.590805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.623684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 413/812] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:32.624735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.708281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.715689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3b75a7-4509-4cfe-9456-6cb4ddf1b47d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.748904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a583d-3544-4437-b9c5-8793973158c0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.755173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.755825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.756102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.756372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.757051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.758561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.785651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 400/813] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:32 2022] POST /identity/v3/endpoints => generated 346 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:32.786551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.795382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.796496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.796982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.797486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.798364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:32.799071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.800832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.802895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:32.816909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 414/814] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:32.822829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.908697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae03cff-02a1-469d-8bab-9ca99a314b19 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.952960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:32.953633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:32.953918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:32.954193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] PATH_INFO: `/v3/services/network` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:32.954808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:32.956268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:32.957634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b1246-da3d-4476-9f9a-b36ee0513019 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:32.971723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:32.972755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 09 04:26:32.973590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:32.976104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 401/815] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/services/network => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:32.977215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:32.996308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:33.009728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.010755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.011265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.011708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.020131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.021808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:33.034595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:33.035606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:33.036522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 415/816] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:33.037475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.105558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:33.108692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f040fd8-6865-48c1-8a13-882fd73b9a0d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.145711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.146345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.146625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.146925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.158349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:33.163736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:33.164667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:33.165505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:33.169817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 402/817] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:33.170744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.171073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427e81b7-9b58-4a8c-bccb-f8a6d42e8731 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.210463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.211120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.211406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.211746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.212422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] RBAC: Adding query filter params (name=network) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:33.212791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.214390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.224631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 416/818] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:33 2022] GET /identity/v3/services?name=network => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:33.225568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.281506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:33.307841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac85a54-0eff-407f-bd3d-59ffdae089c3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.343000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.343612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.343893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.344167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.344860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:33.345167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.346594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.360054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 403/819] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:33.361104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.364278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52e53a-300d-44e9-ba71-4241580ec0cd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.390264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:33.405279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.406425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.406425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.406593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.407211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:33.407662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.409266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.424745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 417/820] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:33.425627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.470598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0754e-1e6b-4ff5-a1f5-0293c47a99c6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.522016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.522537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.522811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.523123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.523758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] RBAC: Adding query filter params (type=network) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:33.524062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.525924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.527357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:33.536275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 404/821] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:33 2022] GET /identity/v3/services?type=network => generated 350 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:33.536275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.547613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb564ef-76e7-410a-a87c-68b2c4f06e53 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.596345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.597569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.598343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.599267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/6a835e652fcc43ab923c9c6ab434d208/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.602198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=6a835e652fcc43ab923c9c6ab434d208, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.604978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.623474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 418/822] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:33 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/6a835e652fcc43ab923c9c6ab434d208/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:33.624592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.678186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f553a162-aaa4-4bcf-b45e-78ac2ae09cff None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.683381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:33.762199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5cc559-c85a-480e-a24a-26e53bfd719f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:33.762840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.763616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.763967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.764304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.791452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.792955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.808221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:33.809261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:33.809748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:33.810235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:33.811847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:33.813550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:33.828800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:33.829889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:33.830805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:33.830805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:33.830805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:33.838527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 405/823] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:33.840051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:33.856563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 419/824] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:33 2022] POST /identity/v3/endpoints => generated 337 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:33.911942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:34.095521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab4c69f-6796-4071-b4ff-c67777ecaa1e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:34.141604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:34.142292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:34.142539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:34.142731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:34.143404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:34.143697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:34.145090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:34.158126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 406/825] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:34.164023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:34.283192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:34.291082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b76cd-4bbe-4f3c-8894-45cdaa4f39d4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:34.328536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:34.329170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:34.329448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:34.329722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:34.337520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:34.339560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:34.350860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:34.351841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:34.352795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 420/826] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:26:34 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:34.360036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:34.466366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dad701d-45a4-4969-bbc7-1e19930d0a9e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:34.473494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:34.507809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:34.508419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:34.508695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:34.508963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:34.509789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:34.510091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:34.511564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:34.533664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 407/827] 10.222.0.20 () {56 vars in 1149 bytes} [Fri Sep 9 04:26:34 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:34.540234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:34.648452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340865f-66aa-4b6e-879a-106e0525b284 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:34.684635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:34.685644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:34.686153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:34.686646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:34.699105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:34.700856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:34.716242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:34.717209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:34.718124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:34.718124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:34.718124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:34.723753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 421/828] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:34.726163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:34.833301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983303bc-9cdc-4efd-ad88-62807d49e1d1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:34.872413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:34.873368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:34.873884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:34.874420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:34.875391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:34.875917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:34.877475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:34.894508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 408/829] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:34.896184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:34.908983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:35.004378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79cefcf-810b-431e-a75b-2e6f7ca0df45 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:35.038683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:35.039347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.039628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.039901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.047120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:35.048720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-9a0c7e5d-47cc-43f9-9db0-ed46b209a7e9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:35.062824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:35.063844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:35.064962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 422/830] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:35.072187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:35.106544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:35.177951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f5ac4-e1d6-4fd3-a8d0-2c56add10d88 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.178268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f5ac4-e1d6-4fd3-a8d0-2c56add10d88 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.178493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f5ac4-e1d6-4fd3-a8d0-2c56add10d88 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.179938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 423/831] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:35.186113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7f103d-58c6-42ba-9c1f-e8c3bdc27f02 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.186238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7f103d-58c6-42ba-9c1f-e8c3bdc27f02 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.186503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7f103d-58c6-42ba-9c1f-e8c3bdc27f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.192307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16d4d9f-06fa-4288-84a5-ab9ef9cf945c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:35.206274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-df7f103d-58c6-42ba-9c1f-e8c3bdc27f02 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:35.207527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-df7f103d-58c6-42ba-9c1f-e8c3bdc27f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:35.228962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:35.229579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.229848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.230120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.230833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:35.231167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:35.235394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f00d708-b86c-4671-a337-67c445ca7d20 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:35.248417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 409/832] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:35.252803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67041ef-1d23-4b48-932d-b149ea7f75cc None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.253316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67041ef-1d23-4b48-932d-b149ea7f75cc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.253683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67041ef-1d23-4b48-932d-b149ea7f75cc None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.256213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 410/833] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:35.256213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67041ef-1d23-4b48-932d-b149ea7f75cc None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:35.283635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:35.289455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 424/834] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:35 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:35.290687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.291122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.291314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.310331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:35.310822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:35.371915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:35.391093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 425/835] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:35 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:35.395525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67041ef-1d23-4b48-932d-b149ea7f75cc None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:35.436784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:35.437368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.437647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.437920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/719e5e2221e04ce0aa7f303b2ea84b89/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.452584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=719e5e2221e04ce0aa7f303b2ea84b89, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:35.454017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:35.467950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 411/836] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:35 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/719e5e2221e04ce0aa7f303b2ea84b89/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:35.576706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:35.675026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:35.685804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2f286f-e4ab-4240-93a8-e244074cff45 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:35.716468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:35.725377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:35.726310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.726810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.727318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.728085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:35.730162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:35.739881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:35.741080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:35.741975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:35.742869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 426/837] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:35 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:35.749038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:35.819519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b53c70-9d58-4a89-a1ae-acc13d798890 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:35.856844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:35.857477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.857752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.858018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.858592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:35.860010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:35.861920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:35.869515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af66ed-8af7-419b-a24a-61a3388afb02 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:35.874806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:35.875968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:35.876903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:35.879712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 412/838] 10.222.0.20 () {56 vars in 1132 bytes} [Fri Sep 9 04:26:35 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:35.884764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:35.916426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:35.918308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:35.918741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:35.919353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:35.920494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:35.921221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:35.923290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:35.935411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 427/839] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:35 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:35.937894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:36.012288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:36.018795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbb770-a8b4-4820-80ee-20140493f302 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:36.066757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:36.067403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.067677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.067941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.068911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:36.069226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:36.071023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:36.078548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee22c1-dbf1-40cf-9785-5b75b05a2b0e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:36.081939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 413/840] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:35 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:36.083635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:36.126103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:36.127139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.127624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.128355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.130092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:36.130609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:36.132307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7993d34a-ae37-4084-92cd-a42f78f73d94 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:36.147405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 428/841] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:35 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:36.152527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a074d6f-a8d0-4983-964f-62e6827feb9e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.152816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a074d6f-a8d0-4983-964f-62e6827feb9e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.153096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a074d6f-a8d0-4983-964f-62e6827feb9e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.155077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 429/842] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:26:36.160220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.160641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.161014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.180187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:36.180802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:36.182169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:36.221168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abfaf8c-b29e-4664-ba00-b816fa949b85 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:36.275867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 430/843] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:36 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:36.275976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:36.275976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.275976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.275976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.276318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:36.276318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:36.276905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:36.282478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 414/844] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:36 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:36.540493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:36.594428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:36.649720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68af162-a84c-4a9f-acf9-0f437994b292 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:36.685916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:36.686895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.687401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.687875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.698544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:36.703584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:36.704481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:26:36.705308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:36.707844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 431/845] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:36.714334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:36.821547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e8dadd-3cc1-4120-bf0e-cd254ab140dd None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:36.847200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:36.856869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:36.857494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:36.857771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:36.858044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:36.858724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:36.859053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:36.860440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:36.869850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 415/846] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:36.875846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:36.979682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ff8ad-74cc-4bff-a734-590e543464b3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:36.991132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:37.019103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:37.019738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.020016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.020287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.031835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:37.033290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:37.048399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:37.049348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:37.050214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:37.050214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:37.050214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:37.050214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 432/847] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:37.056635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:37.168910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c38d42-c075-4a57-8637-ec5889580e2d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:37.203419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:37.204143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.204576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.204856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.205611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:37.205988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:37.207617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:37.222157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 416/848] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:37.228035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:37.333123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df944b3-9d54-4c4d-a01a-593446ec1c42 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:37.367119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:37.367747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.368025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.368296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.376579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:37.378043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:37.389406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:37.390317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:37.391283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 433/849] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:26:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:37.396792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:37.448214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:37.501938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d819793-7650-4ad0-bf97-51a39058f0c9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:37.537642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:37.538250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.538528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.538799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.539555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:37.539853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:37.541343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0107f9-c04a-4136-9312-4a81122ab4c7 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:37.563107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 417/850] 10.222.0.20 () {56 vars in 1135 bytes} [Fri Sep 9 04:26:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:37.568448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:37.595403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:37.677401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0d693c-e740-44ed-921c-085dbc2faaa4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:37.711887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:37.712436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.712803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.713079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.763184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccc9d74-809c-418d-aa33-bb31d88be106 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.763501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccc9d74-809c-418d-aa33-bb31d88be106 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.763747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccc9d74-809c-418d-aa33-bb31d88be106 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.765380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 418/851] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:37.770269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:37.770559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:37.770959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:37.784191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:37.785641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:37.789756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:37.791470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:37.823263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:37.866969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 419/852] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:37.906120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 434/853] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:37 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:26:37.911784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:37.949671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.021853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980b06bb-be35-4d83-afc6-eddfdc6ff1af None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.057606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.058292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.058573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.058844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.069288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.070744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:38.085932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:38.086953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:38.088394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:38.088394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:38.088394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:38.088394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 420/854] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:38.096038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.145992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.204225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b40c202-8e80-45ff-a3d5-09acd384c238 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.229695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.248474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.249250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.249623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.249991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.250733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:38.251273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.252947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.270174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.273311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 435/855] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:38.274241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.288134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9395e61e-1e08-4ffd-92b9-c026397f2b13 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.326239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.327140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.327519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.327888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.341927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:38.348006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:38.349246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:26:38.350329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:38.352020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 421/856] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:38.358414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.412323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edf071c-1173-4905-a995-0d4220e62fb0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.450628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.451104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.451386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.451664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.458613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.460086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-160cc02f-1b40-4638-9071-d059be91c89e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:38.471741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:38.472676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:38.473859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 436/857] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:38.474459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccc3da3-f9d7-4ba7-8256-3259dc3a4f6b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.474459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccc3da3-f9d7-4ba7-8256-3259dc3a4f6b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.474605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccc3da3-f9d7-4ba7-8256-3259dc3a4f6b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.475950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 437/858] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:38.479033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccc3da3-f9d7-4ba7-8256-3259dc3a4f6b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.481922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a7664-a272-4ca7-afe1-0fde4ee3fe14 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.520913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.521522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.521805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.522079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.522975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:38.523088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.524492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3d680a-f7e1-4aa4-8845-3915512ef55b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.531709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.533833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 422/859] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:38.535142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.535429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.535701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.555093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:38.556258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:38.560422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.600256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccc3da3-f9d7-4ba7-8256-3259dc3a4f6b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.629990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 423/860] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:38 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:38.630820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.637544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.638153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.638448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.638996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.639660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:38.639972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.641520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8f8148-74d1-42b7-919e-d24cbbcc84bf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.659693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 438/861] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:38.659693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5499e-d3a4-4d42-b6ad-940e2cc4517c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.659693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5499e-d3a4-4d42-b6ad-940e2cc4517c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.659693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5499e-d3a4-4d42-b6ad-940e2cc4517c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.659693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 439/862] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:38.663176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5499e-d3a4-4d42-b6ad-940e2cc4517c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.733742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240e6be4-b7ef-4a23-ab87-5932a1946049 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.766700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.767349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.767624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.767888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.775997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.777292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5499e-d3a4-4d42-b6ad-940e2cc4517c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:38.778425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.779510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:38.779918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8965f9a0-2cf9-4b08-b7f5-6fda5cc45e2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:38.795002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:38.795967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:38.796922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:38.796922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:38.796922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:38.796922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 424/863] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:38.797637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.797637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.797762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.815880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:38.817047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:38.823008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:38.823613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:38.823972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:38.824371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:38.825097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=6a835e652fcc43ab923c9c6ab434d208) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:38.825405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:38.826949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:38.843078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 440/864] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/role_assignments?user.id=6a835e652fcc43ab923c9c6ab434d208&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:38.843078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.899497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 425/865] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:38 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:38.901028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:38.969922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1303ea7-4373-45ea-b5ca-c36b18546bc2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.018277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.019227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.019658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.020083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.020880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:39.021888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:39.023747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:39.038222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bfae05-b797-4731-9505-23d7bfd7ac7d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.042667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 441/866] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:39.051908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:39.063436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:39.081226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.081945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.082364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.082640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.083331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:39.084708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:39.085383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:39.104436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 426/867] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:26:38 2022] POST /identity/v3/services => generated 251 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:39.180611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e2a324-a4a2-44ff-b520-b9adea89edf1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.196268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:39.242844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.244746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.245991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.246626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.257658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:39.259642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:39.276595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:39.277818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:39.279040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:39.279926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 442/868] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:26:39 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:39.287347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:39.345351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066e393-b245-4655-9dd6-ae9f353d7679 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.390957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.390957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.390957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.390957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.391365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:39.391442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:39.391717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:39.411050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 427/869] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:26:39 2022] POST /identity/v3/services => generated 269 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:39.411809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f55665-ce4c-4063-8d6c-925dbf3465a4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.458491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.459193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.459473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.459744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.460477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:39.460781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:39.462227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:39.483742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 443/870] 10.222.0.20 () {56 vars in 1149 bytes} [Fri Sep 9 04:26:39 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:39.489333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:39.543350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:39.617225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da096d4-f676-4992-a756-29da5a03f355 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.656329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.657324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.657811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.658286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.733661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:39.735248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:39.779060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:39.842212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 428/871] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:39 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:26:39.847930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:39.952534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f704b-d1b5-4473-95cb-2984aa0e0d46 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:39.987645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:39.988276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:39.988554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:39.988827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:39.991580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:39.999539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:40.001224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:40.016688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:40.017634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:40.019154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:40.019154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:40.019154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:40.019154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 444/872] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:40.023417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:40.130663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4844f-7c82-410c-a96c-1be27ef8d49d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:40.166652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:40.167290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:40.167568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:40.167838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:40.168481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:40.168781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:40.170145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:40.182623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 429/873] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:40.188583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:40.297027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd08376-1a57-461e-be69-ecdd159feeaa None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:40.331270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:40.331896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:40.332220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:40.332497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:40.339935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:40.341368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:40.353612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:40.354530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:40.355525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 445/874] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:40.362907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:40.367276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:40.469867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bbd62-3219-4011-bb63-568647c84837 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:40.504410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:40.505344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:40.505845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:40.506312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:40.507223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:40.507722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:40.509446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc06a7cd-99ae-42f8-a396-702587cc907a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:40.522092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 430/875] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:40.528941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:40.582044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:40.634201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a43cded-e491-4729-af95-2fddbacdcb4a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:40.668062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:40.668664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:40.668942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:40.669210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:40.669927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=719e5e2221e04ce0aa7f303b2ea84b89) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:40.670233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:40.671671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539e1ca-2358-471b-96f1-205baa95dfd9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:40.677691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 446/876] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:40 2022] GET /identity/v3/role_assignments?user.id=719e5e2221e04ce0aa7f303b2ea84b89&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:41.221115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c41fd1-5a7b-493a-a052-8a6881753719 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.221497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c41fd1-5a7b-493a-a052-8a6881753719 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.221672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c41fd1-5a7b-493a-a052-8a6881753719 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.223283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 431/877] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:41.228852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5092e-81a2-4bf0-858c-ae05de867962 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.229739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5092e-81a2-4bf0-858c-ae05de867962 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.230214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5092e-81a2-4bf0-858c-ae05de867962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.249533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a6c5092e-81a2-4bf0-858c-ae05de867962 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:41.251124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a6c5092e-81a2-4bf0-858c-ae05de867962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:41.327200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 447/878] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:41 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:41.430369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47719da7-79c9-4277-8f40-c0eb6e4176e3 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.430895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47719da7-79c9-4277-8f40-c0eb6e4176e3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.430895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47719da7-79c9-4277-8f40-c0eb6e4176e3 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.432976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 432/879] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:41.438408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c450b-4631-438b-921e-41e83e04e072 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.438906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c450b-4631-438b-921e-41e83e04e072 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.439394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c450b-4631-438b-921e-41e83e04e072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.458316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dd1c450b-4631-438b-921e-41e83e04e072 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:41.459802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dd1c450b-4631-438b-921e-41e83e04e072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:41.496394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:41.535055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 448/880] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:41 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:41.612310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47719da7-79c9-4277-8f40-c0eb6e4176e3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:41.678722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:41.696368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ef843-ebf9-4c9f-99f1-dfe7603e8953 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.697450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ef843-ebf9-4c9f-99f1-dfe7603e8953 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.697736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ef843-ebf9-4c9f-99f1-dfe7603e8953 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.699356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 449/881] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:41.708317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.708603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.708878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.724296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47719da7-79c9-4277-8f40-c0eb6e4176e3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:41.734425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:41.736389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:41.768693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:41.769963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.770452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.771156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.771892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:41.774251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:41.784964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:41.786167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:41.787280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:41.790457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 433/882] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:41.795659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:41.823952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 450/883] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:41 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:41.824851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:41.906045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37543403-b85b-4408-a79f-edb4c7b776e3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:41.932737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:41.943460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:41.944140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:41.944738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:41.945286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:41.946084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:41.946727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:41.948908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:41.950507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd63d8f4-69c9-4c36-8427-630c63122543 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:41.952541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:41.960814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 434/884] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:41.961892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:41.998609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.000104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.000537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.000972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.001763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.003350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:42.013667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:42.014882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:42.015802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:42.018075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 451/885] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:41 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:42.024018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.080223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b423a7-b3d4-47db-b017-c5b52370f429 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.115718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.116562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.116951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.117288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.118308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.119226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.121152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.126139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 435/886] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:41 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:42.126922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.169051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721280cc-73b6-4f8c-9ee6-8c7f045833ad None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.208660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.209283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.209561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.209836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.210479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.210825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.211335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:42.213120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.223766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 452/887] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:42.224967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.269006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:42.285603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e2bf77-a902-4c91-988b-45bb3c6ab6fb None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.327980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.327980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.327980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.327980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.328429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.329387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:42.339477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:42.340431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:42.342004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 436/888] 10.222.0.20 () {56 vars in 1132 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:42.346508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.377448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74c1cc-1481-44f5-81c9-2db5ece98b8c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.425822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.426597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.426963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.427315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.428674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.429344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.433126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.446179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 453/889] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:42.447803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.469658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2739f5-85d3-40c2-aec2-c66248810b16 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.489794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:42.508957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.510442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.511011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.511501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.512369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.512865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.514437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.524414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 437/890] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:42.525762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.562486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:42.570258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc04f520-da2a-4e10-8cfe-7b0d4e401a85 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.618098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.618756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.619069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.619346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.620053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] RBAC: Adding query filter params (interface=public, service_id=756284e301f84623aad6042450b38459, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.620363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.621896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.631896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 454/891] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/endpoints?service_id=756284e301f84623aad6042450b38459&interface=public®ion_id=RegionOne => generated 194 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:42.634158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.642274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9f23d7-7281-4494-8075-74a68eb2f757 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.687174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.688809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.689325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.689798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.691392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.691909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.694037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.701613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 438/892] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:42.710975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.753237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eedde6f-fa3a-4a4e-9ed2-e2e34ab8339e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.763931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:42.792247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.792860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.793140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.793414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.794025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.795422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.806969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 455/893] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services => generated 1268 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:26:42.832691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537b2d1-a870-43a2-a4f5-f7668b3add3a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:42.875849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:42.876523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.876803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.877127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.877918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] RBAC: Adding query filter params (interface=public, service_id=1061143cb01d436ea9d9e857a44f1223, region_id=RegionOne) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:42.878231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:42.879659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3351d45-7e32-4706-8558-f3f8e5372911 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:42.889043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 439/894] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/endpoints?service_id=1061143cb01d436ea9d9e857a44f1223&interface=public®ion_id=RegionOne => generated 194 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:42.896189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:42.925312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:42.946522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdd544e-5fbc-44e7-9b83-69d68da89ef6 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.947718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdd544e-5fbc-44e7-9b83-69d68da89ef6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.948007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdd544e-5fbc-44e7-9b83-69d68da89ef6 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.949810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 440/895] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:42.956384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:42.956803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:42.956978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:42.976612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:42.977884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:43.020377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ab351-da73-431f-a847-b14f6e66b5dc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:43.055893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 441/896] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:42 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:43.061550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:43.062217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:43.062506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:43.062779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:43.063457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:43.064849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:43.072958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 456/897] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:42 2022] GET /identity/v3/services => generated 1268 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:26:43.337392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:43.379776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:43.441160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ae18f-c177-4e77-a166-5bb0c89476fc None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:43.479573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:43.480214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:43.480466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:43.480741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:43.491175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:43.496605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:43.497512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:26:43.498331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:43.502762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 442/898] 10.222.0.20 () {56 vars in 1120 bytes} [Fri Sep 9 04:26:43 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:43.504780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:43.595836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:43.612316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c92bb-9188-4fae-bdcf-760ee3b8ed1b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:43.647561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:43.648512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:43.649016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:43.649488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:43.650862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:43.651388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:43.652951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:43.661946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 457/899] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:26:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:43.668037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:43.770836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:43.774821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be68b3e-03ff-4e91-af05-d38450bac032 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:43.811176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:43.811804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:43.812136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:43.812408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:43.823103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:43.824558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:43.839327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:43.840291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:43.841166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:43.841166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:43.841166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:43.845117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 443/900] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:43.846577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:43.956429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f896c2-f70d-4bc2-8e87-7980dbd26e79 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:43.997663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:43.998274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:43.998552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:43.998823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:43.999512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:43.999811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:44.001163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:44.013725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 458/901] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:44.020260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:44.125840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e3132-5af6-4f34-a36a-c2acd328ecad None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:44.159632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:44.160252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.160530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.160797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.169262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:44.170704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:44.182713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:44.183674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:44.184983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 444/902] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:26:44 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:44.192019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:44.204065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:44.320867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748debbb-f4eb-4579-b817-e8c46367e868 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:44.358211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:44.358824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.359130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.359400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.360133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:44.360425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:44.361869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdfeeea-8029-4e11-bbaf-adda1465cb39 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:44.380019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 459/903] 10.222.0.20 () {56 vars in 1149 bytes} [Fri Sep 9 04:26:44 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:44.387747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:44.399207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0865fda-0ab0-49b8-81d0-858ed6253b1b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.399495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0865fda-0ab0-49b8-81d0-858ed6253b1b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.400327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0865fda-0ab0-49b8-81d0-858ed6253b1b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.401900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 460/904] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:26:44.406906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.407218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.407493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.408976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:44.428916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:44.430218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:44.499543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:44.515428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 461/905] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:44 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:44.518396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec4cf78-1038-4cdb-a79d-415d4323d4dd None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:44.552197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:44.553100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.553581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.554051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.623352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:44.624798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:44.639388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:44.732077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 445/906] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:44 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:26:44.737648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:44.790639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:44.843052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05966d37-1ec6-4d98-9381-c1e092e89330 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:44.878624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:44.879459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.879735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.880085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.890901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:44.892403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:44.902016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2360b522-1060-42f1-ad84-7c9efe18f50f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:44.908883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:44.909891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:44.910795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:44.910795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:44.910795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:44.914698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 462/907] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:44.916005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:44.920363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:44.946096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:44.947225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:44.947685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:44.948179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:44.949016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:44.950695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:44.969154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 446/908] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:26:44 2022] POST /identity/v3/services => generated 241 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:26:44.969868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:45.036129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbb8cd2-3ba7-4df8-a8ce-d284cb1cb9f0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.082719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:45.083452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.083738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.084013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.084665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:45.084973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:45.086354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec1e4bb-ba5d-43e8-b4de-5ca5e148c3c0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:45.106817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 463/909] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:45.108287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:45.228290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc588db-455c-4d89-bb85-24ad59748d67 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.267741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:45.268385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.268666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.268942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.276340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:45.277810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:45.278813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287119c-8fde-47b6-9445-f5e826f0f2cd None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.279179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287119c-8fde-47b6-9445-f5e826f0f2cd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.279459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287119c-8fde-47b6-9445-f5e826f0f2cd None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.280922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 464/910] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:45.286248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9cd1a9-32c2-4a16-8ce1-e86e63b07c52 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.286764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9cd1a9-32c2-4a16-8ce1-e86e63b07c52 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.287275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9cd1a9-32c2-4a16-8ce1-e86e63b07c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:45.293376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:45.294367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:45.295312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:45.299030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 447/911] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:45.300938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:45.313106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9f9cd1a9-32c2-4a16-8ce1-e86e63b07c52 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:45.314401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9f9cd1a9-32c2-4a16-8ce1-e86e63b07c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:45.333278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:45.395093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 465/912] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:45.397661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7247ff0-a7ca-42d5-8684-db3bb9648bd1 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.398270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7247ff0-a7ca-42d5-8684-db3bb9648bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.398777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7247ff0-a7ca-42d5-8684-db3bb9648bd1 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.401443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 466/913] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:45.406946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.407253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.407530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.412619 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab09d3e-2751-4b8a-97ab-25b022084799 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.435552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:45.437116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:45.451371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:45.451980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.452251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.452525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.453331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:45.453630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:45.455087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:45.473773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 448/914] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:45.474515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:45.475808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:45.531329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 467/915] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:45.577337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:45.588317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6395f372-feca-4856-bf6a-6e15aac8454b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.625000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:45.625604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.625937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.626214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.626957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] RBAC: Adding query filter params (role.id=75781bdd4b9e45a49f9e77b94ad8b0e6, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=719e5e2221e04ce0aa7f303b2ea84b89) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:45.627274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:45.628753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:45.634695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 449/916] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:45 2022] GET /identity/v3/role_assignments?user.id=719e5e2221e04ce0aa7f303b2ea84b89&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=75781bdd4b9e45a49f9e77b94ad8b0e6 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:45.653266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:45.796390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ddba10-f219-49f9-9469-31f781a1d9b9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.815170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:45.839682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:45.840351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.840726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.841004 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.841599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:45.843006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:45.854166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:45.855430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:45.859396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 468/917] 10.222.0.20 () {56 vars in 1132 bytes} [Fri Sep 9 04:26:45 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:45.864081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:45.895347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:45.941626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd2c1ce-a520-4bcc-ab57-7232eec5d297 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.976100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a720145-1a7f-4571-a853-a995c4062d93 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:45.990235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:45.991622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:45.992104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:45.992905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:45.994046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:45.995975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:46.007930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:46.008912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 09 04:26:46.010275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:46.013001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 450/918] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:45 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:46.018893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:46.027916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:46.028526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:46.028818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:46.029071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:46.029697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:46.030011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:46.031941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.042897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 469/919] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:45 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:46.043946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:46.049568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:46.143966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:46.147083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb8811-ebb5-401a-bfe8-f53ff8943819 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:46.155677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d14784-dc3a-426d-afee-2aebf9df904c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:46.195750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:46.196361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:46.196670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:46.196943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:46.197578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:46.197881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:46.199265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.199869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:46.200805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:46.201386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:46.201848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:46.202673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:46.203209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:46.204584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 470/920] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:46 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:46.209785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:46.210516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.217147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 451/921] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:46 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:46.222666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:46.326918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b2599-0b7b-49a6-bc39-3092fa2610b2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:46.335309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:46.346038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a80a5d4-7fef-462a-bd6e-a5263d457d8c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:46.364674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bea9ac-34c1-4216-82b3-e7c834639c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:46.365427 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bea9ac-34c1-4216-82b3-e7c834639c08 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:46.365931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bea9ac-34c1-4216-82b3-e7c834639c08 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:46.366300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bea9ac-34c1-4216-82b3-e7c834639c08 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:46.367397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bea9ac-34c1-4216-82b3-e7c834639c08 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:46.369409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bea9ac-34c1-4216-82b3-e7c834639c08 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.385550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:46.386233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:46.386514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:46.386847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:46.387605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:46.388133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:46.389603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.394966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 452/923] 10.222.0.20 () {56 vars in 1114 bytes} [Fri Sep 9 04:26:46 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:46.395181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 471/923] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:46 2022] POST /identity/v3/endpoints => generated 347 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:46.405353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:46.521175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d47e5a-9e52-4c8a-8897-c832b15c5838 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:46.569105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdaa223-c45e-4bea-8e31-8372b630a821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:46.570238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdaa223-c45e-4bea-8e31-8372b630a821 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:46.570765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdaa223-c45e-4bea-8e31-8372b630a821 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:46.571270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdaa223-c45e-4bea-8e31-8372b630a821 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:46.573241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdaa223-c45e-4bea-8e31-8372b630a821 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:46.575102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdaa223-c45e-4bea-8e31-8372b630a821 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:46.594212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:46.603196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 453/924] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:46 2022] POST /identity/v3/endpoints => generated 346 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:47.311811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91fe3b-d7af-4525-9e9d-a1c5f3b9fd8a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:47.312335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91fe3b-d7af-4525-9e9d-a1c5f3b9fd8a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:47.312335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91fe3b-d7af-4525-9e9d-a1c5f3b9fd8a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:47.313947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 472/925] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:47.319586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:47.319875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:47.320151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:47.340106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:47.341344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:47.415281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 454/926] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:47.695697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91fe3b-d7af-4525-9e9d-a1c5f3b9fd8a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:47.711840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:47.809182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91fe3b-d7af-4525-9e9d-a1c5f3b9fd8a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:47.844127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:47.845115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:47.845597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:47.846091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:47.846957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:47.848537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-9be40dc0-7f11-4626-9270-8d3769670b44 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:47.857858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:47.858850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:47.859720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:47.862364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 473/927] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:47 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:47.869191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:47.872947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691b786-bdd8-454d-9c9f-e1d263eb6afc None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:47.873234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691b786-bdd8-454d-9c9f-e1d263eb6afc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:47.873508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691b786-bdd8-454d-9c9f-e1d263eb6afc None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:47.874971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 474/928] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:47.880817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:47.881099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:47.881370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:47.903804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:47.905278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:47.983011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ff2263-c2c1-43ca-888b-f5a952f098ea None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:47.995010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 475/929] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:48.017117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:48.026754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.027823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.028360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.028839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.029684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:48.030239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:48.031870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:48.037138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 455/930] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:47 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:48.043132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.059502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:48.146712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98b055b-386b-4b58-a26b-6ddab70d80e9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:48.185837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.186446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.186726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.187017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.187645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:48.187973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:48.189296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:48.194547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 476/931] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:48.200768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.305543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f62f98-1dd5-43d8-819f-2fc07922a073 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:48.306714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.346172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.346782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.347096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.347371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.348248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] RBAC: Adding query filter params (interface=public, service_id=74cd1a70eb674e93a29b4f0178edcd2a, region_id=RegionOne) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:48.348558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:48.350582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:48.361490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 456/932] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/endpoints?service_id=74cd1a70eb674e93a29b4f0178edcd2a&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:48.367066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.393530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:48.427648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:48.430294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c3679b-3d3f-46db-8a24-ea62e2a49bb8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:48.472079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.472715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.472995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.473267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.481496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbb71d8-7e81-49f8-98b9-825ba44c909b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:48.486479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:48.491514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:48.492425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:26:48.493353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:48.497812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 477/933] 10.222.0.20 () {56 vars in 1120 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:48.499085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.529679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.530858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.531341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.532039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.533231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:48.535064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:48.549196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 457/934] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/services => generated 1497 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:26:48.621561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae69c429-e5b7-4765-ba53-13c59a4fe291 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:48.655098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:48.665356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.666513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.667054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.667564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.668462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:48.668982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:48.670696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb83d9c-b20f-49d3-906a-fae521dc0fd3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:48.681902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 478/935] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:48.687796 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.694142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983c7398-3240-490e-8bc8-8d518504bdc6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.694779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983c7398-3240-490e-8bc8-8d518504bdc6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.695340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983c7398-3240-490e-8bc8-8d518504bdc6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.698581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 479/936] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:26:48.704027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1486c85b-fd90-4f11-9e2f-3e7176b66366 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.704230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1486c85b-fd90-4f11-9e2f-3e7176b66366 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.704446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1486c85b-fd90-4f11-9e2f-3e7176b66366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.727806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1486c85b-fd90-4f11-9e2f-3e7176b66366 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:48.729094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1486c85b-fd90-4f11-9e2f-3e7176b66366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:48.813536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 480/937] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:48.821885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e8b91-5d07-439d-a8e5-7fef01d5df56 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:48.828525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b538fe-76d0-4915-96c9-f026a242eb67 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.828814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b538fe-76d0-4915-96c9-f026a242eb67 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.829085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b538fe-76d0-4915-96c9-f026a242eb67 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.830402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 481/938] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:26:48.835397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.835682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.835955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.843324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:48.856296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:48.857453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:48.862080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:48.862983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:48.863562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:48.864063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:48.880163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:48.881921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:48.898984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:48.900737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:48.901837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:48.901837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:48.901837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:48.903161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 458/939] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:48.908956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:48.946956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 482/940] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:49.021572 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60af77d-4f90-4c04-8914-dec7edfc42e0 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.056787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.057647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.057923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.058193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.058833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:49.059163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.060640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.073774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 459/941] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:49.079228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.084610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.116473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.153301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.187947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce7fcac-7a46-41f8-a2e2-ad14804f9764 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.217012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62112c9c-1bdc-435b-9c02-5979cb599fd0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.225101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.225727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.226002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.226271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] PATH_INFO: `/v3/services/compute` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.226855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.228770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:49.237779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:49.239051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:49.240280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.243107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 460/942] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/services/compute => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:49.244940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.265278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.266040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.266484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.266653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.275172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.276693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:49.288291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:49.289236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:49.290778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:49.294853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 483/943] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:49.295779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.350480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.358173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956052c9-32bb-4f49-84c8-d95fb652b4fe None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.393249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.393974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.394251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.394523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.395296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.396730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.396830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1767b8-6b77-4876-b09e-5f4e50dd75c0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.400148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING keystone.server.flask.common [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 09 04:26:49.411781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.418686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.sql.core [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 09 04:26:49.418686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 09 04:26:49.418686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [parameters: {'id': '50b0873e8a7d4165af9ef910b776289e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 09 04:26:49.418686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81451) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 50b0873e8a7d4165af9ef910b776289e.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 50b0873e8a7d4165af9ef910b776289e. Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application result = self._query(query) Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application conn.query(q) Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 09 04:26:49.435395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application result.read() Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 09 04:26:49.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application next(self.gen) Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application yield resource Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application next(self.gen) Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self.session.rollback() Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self.force_reraise() Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise self.value Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application session.commit() Sep 09 04:26:49.442679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._prepare_impl() Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self.session.flush() Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._flush(objects) Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application compat.raise_( Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application flush_context.execute() Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 09 04:26:49.444160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rec.execute(self) Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 09 04:26:49.445041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application result = self._query(query) Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application conn.query(q) Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application result.read() Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 09 04:26:49.447026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [parameters: {'id': '50b0873e8a7d4165af9ef910b776289e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.449169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:49.451129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 50b0873e8a7d4165af9ef910b776289e. Sep 09 04:26:49.453506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:49.454753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.454753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.454753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.454753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.455349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 461/944] 10.222.0.20 () {60 vars in 1140 bytes} [Fri Sep 9 04:26:49 2022] POST /identity/v3/projects => generated 262 bytes in 193 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:49.455542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:49.455542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.455807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.455947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.455947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 484/945] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:49.455947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.557475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc567b-133a-41e1-930f-87230e6eec59 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.559746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144ea2bd-e676-4b1c-8486-72a5e3be0366 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.597062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.597691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.597961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.598233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.599443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.600163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.600586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.600911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.601735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:49.602144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.604526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.606260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.607700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:49.619681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:49.620773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:49.622244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:49.625587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 485/946] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:49.626399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.632732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 462/947] 10.222.0.20 () {56 vars in 1149 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:49.632732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.644518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.661324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.735191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9f509e-da17-4867-9b05-f53aa99e754d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.741077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405dbeb2-9aed-4894-922c-127fa2e6861a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.771357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.771961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.772240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.772514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.773142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] RBAC: Adding query filter params (type=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:49.773445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.774834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.786965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 486/948] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:49.786965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:49.789959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.791567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.792993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.793348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.794124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:49.794499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.796156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.821958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 463/949] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:49.914272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:49.922095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc489b59-0dbb-46f7-a939-dbaef482d7cc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:49.964379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:49.965034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:49.965315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:49.965587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:49.978801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:49.980321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:49.996663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:49.997893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:49.999181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:49.999181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:49.999181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:49.999181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 487/950] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:50.003837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:50.088482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:50.124597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c5a8f3-20cb-48cb-9a79-784b20cbedb9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:50.160571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:50.161330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:50.161708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:50.162082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:50.162827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:50.163256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:50.164789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:50.176866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 464/951] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:50.183243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:50.298785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d52704-88db-4351-b440-71b3f4b6e287 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:50.336150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:50.337562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:50.338016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:50.338480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:50.346115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:50.347783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:50.359406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:50.360460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:50.361809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:50.368847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 488/952] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:50.370343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:50.477050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca6d5b-3a78-46bb-8a47-615e19d4d79f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:50.504170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:50.513417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:50.514030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:50.514306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:50.514573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:50.515313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:50.515617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:50.517041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b71ae-9f21-4938-8225-5ab1ff0f1d66 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:50.529904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 465/953] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:50.535881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:50.553201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:50.644994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017bc93f-662f-482e-8b94-4b4a5b1e0c27 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:50.680483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68775e9a-2cee-4e07-9252-d55a7f13a2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:50.681551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68775e9a-2cee-4e07-9252-d55a7f13a2a5 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:50.682080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68775e9a-2cee-4e07-9252-d55a7f13a2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:50.682829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68775e9a-2cee-4e07-9252-d55a7f13a2a5 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/719e5e2221e04ce0aa7f303b2ea84b89/roles/75781bdd4b9e45a49f9e77b94ad8b0e6` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:50.685282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68775e9a-2cee-4e07-9252-d55a7f13a2a5 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=719e5e2221e04ce0aa7f303b2ea84b89, role_id=75781bdd4b9e45a49f9e77b94ad8b0e6)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:50.687017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68775e9a-2cee-4e07-9252-d55a7f13a2a5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:50.711408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 489/954] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:26:50 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/719e5e2221e04ce0aa7f303b2ea84b89/roles/75781bdd4b9e45a49f9e77b94ad8b0e6 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:26:50.893846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe2a30-4f38-483d-a122-05889c023f27 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:50.894454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe2a30-4f38-483d-a122-05889c023f27 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:50.895001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe2a30-4f38-483d-a122-05889c023f27 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:50.896835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 466/955] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:50.901902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:50.902456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:50.903018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:50.924071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:50.925594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:50.999101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 490/956] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:51.263965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe2a30-4f38-483d-a122-05889c023f27 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:51.368859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe2a30-4f38-483d-a122-05889c023f27 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:51.403884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:51.404508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:51.404786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:51.405054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:51.405628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:51.407018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:51.416313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:51.417713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:51.418815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:51.421623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 467/957] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:51.427296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:51.450765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:51.534757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cdda78-7929-423d-b2d7-6d604d2bc1e2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:51.571830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:51.572688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:51.572861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:51.573026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:51.573660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:51.573963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:51.575361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:51.580582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 491/958] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:51 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:51.585949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:51.651336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:51.708017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fa978-30f0-44d0-942a-43b3881a4f6d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:51.743803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:51.744414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:51.744693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:51.744965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:51.745600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:51.745907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:51.747423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e54d4dd-dfe1-4a74-b914-a8e55a4d57bf None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:51.752784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 468/959] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:51 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:51.758527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:51.830672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:51.867142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24442ee6-baa5-492f-9f11-06f0778fb63a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:51.903082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec40c5f-b0a6-476a-8141-6c9a12375a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:51.903698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec40c5f-b0a6-476a-8141-6c9a12375a81 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:51.903981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec40c5f-b0a6-476a-8141-6c9a12375a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:51.904258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec40c5f-b0a6-476a-8141-6c9a12375a81 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:51.904941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec40c5f-b0a6-476a-8141-6c9a12375a81 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:51.906305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec40c5f-b0a6-476a-8141-6c9a12375a81 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:51.930695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 492/960] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:51 2022] POST /identity/v3/endpoints => generated 349 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:52.093596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29167cc-a803-4407-aab2-ae7d09cd32ff None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.093908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29167cc-a803-4407-aab2-ae7d09cd32ff None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.094071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29167cc-a803-4407-aab2-ae7d09cd32ff None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.095880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 469/961] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:52.104144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a3e24-2ad6-41a2-aed8-2e81c97bc51c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.104492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a3e24-2ad6-41a2-aed8-2e81c97bc51c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.104697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a3e24-2ad6-41a2-aed8-2e81c97bc51c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.125199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-454a3e24-2ad6-41a2-aed8-2e81c97bc51c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:52.126909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-454a3e24-2ad6-41a2-aed8-2e81c97bc51c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:52.204778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 493/962] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:52.209648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a2dfc-382b-4272-ac14-4613bd5625a6 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.209935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a2dfc-382b-4272-ac14-4613bd5625a6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.210213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a2dfc-382b-4272-ac14-4613bd5625a6 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.211969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 470/963] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:52.220129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.220421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.220705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.243728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:52.245251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:52.297603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:52.323582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 494/964] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:52.473576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07a2dfc-382b-4272-ac14-4613bd5625a6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:52.581181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07a2dfc-382b-4272-ac14-4613bd5625a6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:52.598268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:52.605547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:52.617137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:52.617747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.618556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.619079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.620118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:52.621496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:52.639087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 471/965] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:26:52 2022] POST /identity/v3/services => generated 242 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:52.725574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca11385-31e4-4bad-bb76-9b41c7cabb49 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:52.771444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:52.772369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.772793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.773208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.774045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:52.775761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:52.778552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING keystone.server.flask.common [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 09 04:26:52.793186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.sql.core [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 09 04:26:52.793186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 09 04:26:52.793186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [parameters: {'user_id': 'fc82dfb40bc84f82a779f08acef60388', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 09 04:26:52.793186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7caed8df-3af1-4230-9e2e-a1c03b59d1c2 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Sep 09 04:26:52.801967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 09 04:26:52.803147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application yield resource Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.rollback() Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.force_reraise() Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise self.value Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application session.commit() Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 09 04:26:52.804358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._prepare_impl() Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.flush() Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._flush(objects) Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application compat.raise_( Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application flush_context.execute() Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rec.execute(self) Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 09 04:26:52.805554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 09 04:26:52.806675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 09 04:26:52.808021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [parameters: {'user_id': 'fc82dfb40bc84f82a779f08acef60388', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:52.809042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 09 04:26:52.810079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 09 04:26:52.811428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 09 04:26:52.811428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 09 04:26:52.811428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 09 04:26:52.811428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 09 04:26:52.811428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 09 04:26:52.811428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:52.813693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 495/966] 10.222.0.20 () {60 vars in 1131 bytes} [Fri Sep 9 04:26:52 2022] POST /identity/v3/users => generated 160 bytes in 208 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:52.819520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:52.936772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7517045-f4c2-4b45-920b-5a260714e5a6 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:52.973802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:52.974504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.974789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.975095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.982849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8356363-2a03-46ab-87c4-1ab58bbaf6b3 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.983108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8356363-2a03-46ab-87c4-1ab58bbaf6b3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.983379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8356363-2a03-46ab-87c4-1ab58bbaf6b3 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.983666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:52.985276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:52.987859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 496/967] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:52.993086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:52.993380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:52.993659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:52.997773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:52.998911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 09 04:26:53.000115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:53.000663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 472/968] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:26:52 2022] GET /identity/v3/users/cinder => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:53.007105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:53.015623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:53.017144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:53.046774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:53.053211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:53.106615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 497/969] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:52 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:53.134405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb48a181-02f8-4060-a858-011d2b55c2ec None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:53.171649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:53.172591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:53.173106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:53.173602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:53.174516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:53.175068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:53.176763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:53.214174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 473/970] 10.222.0.20 () {56 vars in 1101 bytes} [Fri Sep 9 04:26:53 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:53.390984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:53.499248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0499af8-8b05-4ece-a069-12929d072495 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:53.535524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:53.536501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:53.537011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:53.537497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:53.548437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:26:53.553644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:53.554658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:26:53.556429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:53.557354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 498/971] 10.222.0.20 () {56 vars in 1120 bytes} [Fri Sep 9 04:26:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:53.563738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:53.662169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:53.670872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d1e36a-3ba5-4433-965c-15fbc9564c3c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:53.704986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:53.705594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:53.705874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:53.706151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:53.706824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:53.707164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:53.708531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:53.716665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 474/972] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:26:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:53.722177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:53.751120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:53.823751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2b97fd-c35f-43be-927b-7752e1bd64e1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:53.858787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:53.859451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:53.859734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:53.860007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:53.870087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:53.871678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:53.886204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:53.887193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:53.888075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:53.888075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:53.888075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:53.888810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 499/973] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:53.893053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:53.996526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7d8fac-e831-4b6c-812b-fe491332d5ff None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.033814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.034425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.034705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.035009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.035657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:54.035963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.037345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.052050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 475/974] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:54.058150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.138370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:54.163430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9cacbe-d879-4320-afcc-917bae6f51bf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.198054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.198819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.199226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.199600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.207048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.208583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6aa6faa7-5d77-4e18-adb5-7e1eff2d6df0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:26:54.220215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:54.221350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 09 04:26:54.222593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:54.224202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 500/975] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:54.230293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.249087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:54.254687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b39f6e-6479-4194-af69-97e2dbe1d53a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.255119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b39f6e-6479-4194-af69-97e2dbe1d53a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.255517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b39f6e-6479-4194-af69-97e2dbe1d53a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.256988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 501/976] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:26:54.262580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.262580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.262580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.280928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:54.282103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:54.340770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b1b125-dbc2-48a6-9b64-7075e8f58a94 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.360466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 502/977] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:54 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:26:54.379614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.380366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.380743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.381125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.381947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:54.382355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.384435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.406976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 476/978] 10.222.0.20 () {56 vars in 1149 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:54.408360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.477738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:54.512848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bb04d0-a5f4-48e4-8796-334841d824bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.545921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.546526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.546805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.547121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.616982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.618802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.636283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.699781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:54.720302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:54.738641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 503/979] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:26:54.744029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.750757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d6728-d43a-4186-877f-a804fe13582b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.787057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.787800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.788091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.788362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.789071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.791084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:54.800405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:54.801430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:54.802260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 477/980] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:54.808032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.854046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cbd7eb-3d92-4eab-aa30-919467424bff None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.889316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.890002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.890292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.890617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.901341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.902788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.913856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cd4ba-1736-4b1b-a9f5-644c45931623 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:26:54.918255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:54.919216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:26:54.925131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:26:54.925131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:26:54.925131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:26:54.925131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 504/981] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:54.925131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.947633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:54.953870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:54.954496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.954847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.955215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.955847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:54.956160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:54.957543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef3c3ee-76ad-431f-b329-2d77cc5e294a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:54.962853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 478/982] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:54.964048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f6ab1a-da32-46cb-a02c-50933b9b44f0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:54.964331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f6ab1a-da32-46cb-a02c-50933b9b44f0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:54.964610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f6ab1a-da32-46cb-a02c-50933b9b44f0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:54.965811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 479/983] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:54 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:54.968923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f6ab1a-da32-46cb-a02c-50933b9b44f0 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:54.972011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.029893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdbf7c-ccb3-4a42-8d5f-af948c8cab5a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.065011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.065647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.065920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.066188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.066823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.067153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.068517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f37de-3297-44b8-bbec-b30ddd0fd230 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.073762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f6ab1a-da32-46cb-a02c-50933b9b44f0 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.081332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 505/984] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:55.082497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.082780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.083084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.101420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:55.102657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:55.111414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.112017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.112292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.112561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.113185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.113485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.114835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.119495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 480/985] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:54 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:55.120253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.170836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 506/986] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:55.171925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.183289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.195522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.226956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7833562-9450-4cc5-8167-febdd1ca9653 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.261518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.262151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.262437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.263135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.269617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.271090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:26:55.282200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:55.283286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:26:55.284390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:55.285990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6b5629-16d4-4f94-b49e-b29951614a8c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.288218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 481/987] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:55.289044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.325914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.326561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.326847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.327195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.327898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] RBAC: Adding query filter params (interface=admin, service_id=74cd1a70eb674e93a29b4f0178edcd2a, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.328206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.329586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.339743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 507/988] 10.222.0.20 () {56 vars in 1249 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/endpoints?service_id=74cd1a70eb674e93a29b4f0178edcd2a&interface=admin®ion_id=RegionOne => generated 193 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:55.340713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.398393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf8be15-a777-4eaa-adfe-ffb1964921f1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.417478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.428255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.439098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.439713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.439992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.440268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.440964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.441273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.442755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.446051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff7465-9276-4abe-ab61-52060b3b8cb9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.454623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 482/989] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:55.455448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.481518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.482054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.482329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.482597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.483240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.484579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.490719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 508/990] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:26:55.491628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.562096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b715b342-0a9a-48f0-9937-be8878ca9c0c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.601284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.601925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.602205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.603087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.603685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.605074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:55.614272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:55.615458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:55.616328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 483/991] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:55.621227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11feb56f-01a6-4f61-9ce8-7b9bf8c88d9d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.621875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.660820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.661566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.661853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.662122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.662994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] RBAC: Adding query filter params (role.id=75781bdd4b9e45a49f9e77b94ad8b0e6, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=719e5e2221e04ce0aa7f303b2ea84b89) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.663198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.664619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.672029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 509/992] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/role_assignments?user.id=719e5e2221e04ce0aa7f303b2ea84b89&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=75781bdd4b9e45a49f9e77b94ad8b0e6 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:55.675579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.732317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2452afff-e2ba-43e0-9a2f-8adbec1a5973 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.769265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.769891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.770169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.770438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.771120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.771430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.772822 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.778485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 484/993] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:55.784472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:55.787640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:55.895662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8637b0e-eaa8-45b1-8067-db18e28262dd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:55.935537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:55.936159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:55.936435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:55.936710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:55.937344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:55.937651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:55.939037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:55.944113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 510/994] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:55.950316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:56.053085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd188143-40bc-4e6e-a02f-3b079e947f08 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:56.067816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:56.090120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:56.090724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:56.091030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:56.091303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:56.091997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] RBAC: Adding query filter params (interface=public, service_id=3e82a196416a442398b66893c1518c13, region_id=RegionOne) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:56.092304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:56.093694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0411ac-d1c8-4d0e-a9b0-9ce3faf28f17 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:56.098884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 485/995] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:26:55 2022] GET /identity/v3/endpoints?service_id=3e82a196416a442398b66893c1518c13&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:56.104909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:56.169687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:56.208303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41291d-ca5b-46a9-bace-e131c7ddc96b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:56.245802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533881e3-ab13-4e86-b046-2fc83d95c35f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:56.246405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533881e3-ab13-4e86-b046-2fc83d95c35f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:56.246682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533881e3-ab13-4e86-b046-2fc83d95c35f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:56.246969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533881e3-ab13-4e86-b046-2fc83d95c35f None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:56.247578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533881e3-ab13-4e86-b046-2fc83d95c35f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:56.248935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533881e3-ab13-4e86-b046-2fc83d95c35f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:56.255270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 511/996] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:26:56 2022] GET /identity/v3/services => generated 1727 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:26:57.673699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c930d5-ff0d-4adb-be58-34a61428d6c9 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:57.674221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c930d5-ff0d-4adb-be58-34a61428d6c9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:57.674221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c930d5-ff0d-4adb-be58-34a61428d6c9 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:57.675745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 486/997] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:57.681354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234be41f-f7a3-42e1-9676-cb6ccab9e20f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:57.681761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234be41f-f7a3-42e1-9676-cb6ccab9e20f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:57.681926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234be41f-f7a3-42e1-9676-cb6ccab9e20f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:57.701743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-234be41f-f7a3-42e1-9676-cb6ccab9e20f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:57.702911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-234be41f-f7a3-42e1-9676-cb6ccab9e20f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:57.773252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 512/998] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:57 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:57.876709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea8f5a8-0461-4951-be15-ac8a2b99eeb4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:57.877046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea8f5a8-0461-4951-be15-ac8a2b99eeb4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:57.877266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea8f5a8-0461-4951-be15-ac8a2b99eeb4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:57.878749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 487/999] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:57.883830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:57.884123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:57.884398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:57.903094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:57.904270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:57.959801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:57.972192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 513/1000] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:57 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:58.052930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea8f5a8-0461-4951-be15-ac8a2b99eeb4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.156066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea8f5a8-0461-4951-be15-ac8a2b99eeb4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.188863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.189487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.189763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.190030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.190601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.191986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:58.201080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:58.202094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 09 04:26:58.203337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:58.207209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 488/1001] 10.222.0.20 () {56 vars in 1126 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services/object-store => generated 93 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:58.208423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.244888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.298400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:58.327307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5e96eb-0c55-4469-91a2-810449dcb998 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.362081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467c9760-53aa-4c33-8819-4ec9f87f5701 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.364933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.364933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.365161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.365417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.366045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:58.366348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.368277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.378716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 514/1002] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:58.379589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.413765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.414401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.414742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.415555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] PATH_INFO: `/v3/services/image` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.416144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.417546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ac5b73bb-f9b1-4819-8354-2efedf8c3323 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:58.426248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:58.427443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:58.428467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 489/1003] 10.222.0.20 () {56 vars in 1105 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services/image => generated 86 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:26:58.429297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc58691-f30a-444d-821e-8d7922a3b783 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.429297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc58691-f30a-444d-821e-8d7922a3b783 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.429297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc58691-f30a-444d-821e-8d7922a3b783 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.430467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 490/1004] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:26:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:26:58.433545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc58691-f30a-444d-821e-8d7922a3b783 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.446618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:58.491626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85d65a-d2bd-49c2-9b0f-921a48e2354f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.526963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.527509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.527786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.528055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.528671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:58.528969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.530313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35f341-f0af-42c3-b835-316a72619f26 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.535160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 515/1005] 10.222.0.20 () {56 vars in 1122 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:58.535736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc58691-f30a-444d-821e-8d7922a3b783 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.536357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.536644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.536921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.544140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:58.555242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:26:58.556434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:26:58.573862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.574473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.574754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.575069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.575701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] RBAC: Adding query filter params (name=image) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:58.576004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.577367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.582014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 491/1006] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:58.582765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.623941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 516/1007] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:26:58 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:26:58.625179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.681165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:58.690409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b292e-fa63-479a-969f-8b49b4b866d9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.725748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.726591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.727074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.727342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.728002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.729422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.734411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba04b89-5525-46f0-be75-419b49e54e4e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:58.754966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 492/1008] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:58 2022] POST /identity/v3/endpoints => generated 325 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:26:58.798719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:58.799295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:58.799571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:58.799895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:58.800535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] RBAC: Adding query filter params (type=image) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:58.800836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:58.802388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:58.815286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 517/1009] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services?type=image => generated 122 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:58.898066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:58.934459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:59.014189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a63bcf-9015-4d29-b3f7-13d89675b69b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:59.053946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:59.054506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:59.054787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:59.055101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:59.056251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:59.057876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:26:59.067085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:26:59.068076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:26:59.068910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 493/1010] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:26:58 2022] GET /identity/v3/services/compute => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:26:59.076329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:59.179978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1f2c7e-c6af-414d-880c-05d59c02e74e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:59.190299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:59.217979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:59.218585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:59.218859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:59.219158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:59.219785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:59.220088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:59.221435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:59.231264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 518/1011] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:59 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:26:59.232324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:59.342956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:26:59.350803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60ec23d-ea76-4a69-9cd7-e6de22e2e410 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:59.386797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:59.387438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:59.387720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:59.387994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:59.388625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] RBAC: Adding query filter params (type=compute) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:26:59.388929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:59.390293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a51bcc-9f7c-4d53-9811-7c778e06d29e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:59.399922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 494/1012] 10.222.0.20 () {56 vars in 1112 bytes} [Fri Sep 9 04:26:59 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:26:59.400974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:26:59.512125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0622dfc5-1208-4d24-9915-281391f52020 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:26:59.548489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc6527e-5f06-434a-a48c-010658489063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:26:59.549094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc6527e-5f06-434a-a48c-010658489063 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:26:59.549377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc6527e-5f06-434a-a48c-010658489063 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:26:59.549645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc6527e-5f06-434a-a48c-010658489063 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:26:59.550319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc6527e-5f06-434a-a48c-010658489063 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:26:59.551889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc6527e-5f06-434a-a48c-010658489063 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:26:59.583001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 519/1013] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:26:59 2022] POST /identity/v3/endpoints => generated 334 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:00.963466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dac139-4af8-41bd-a027-dd2bcd0fad3a None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:00.964152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dac139-4af8-41bd-a027-dd2bcd0fad3a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:00.964152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dac139-4af8-41bd-a027-dd2bcd0fad3a None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:00.965530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 495/1014] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:00.970535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:00.971057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:00.971161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:00.989606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:00.990814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:01.015375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:01.020384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2fbd4b-295b-49d5-a180-151f77245d44 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.020672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2fbd4b-295b-49d5-a180-151f77245d44 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.020940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2fbd4b-295b-49d5-a180-151f77245d44 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.022264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 496/1015] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:27:01.027195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.027479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.027753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.045853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:01.047399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:01.073332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 520/1016] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:00 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:01.116088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 497/1017] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:01 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:27:01.337302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:01.395199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:01.441735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37f7fb0-e513-4dcb-8319-9eba4e99c6d3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:01.477327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:01.477961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.478244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.478540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.479806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:01.481260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:01.486976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 521/1018] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:01.488362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:01.492217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:01.507291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536471c5-7735-4118-b0e2-95c35b3854ef None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:01.514130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:01.546878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e18e66-6d96-4cde-8db5-87a0bc38bc6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:01.547518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e18e66-6d96-4cde-8db5-87a0bc38bc6b None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.547793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e18e66-6d96-4cde-8db5-87a0bc38bc6b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.548060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e18e66-6d96-4cde-8db5-87a0bc38bc6b None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.548721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e18e66-6d96-4cde-8db5-87a0bc38bc6b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:01.550333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e18e66-6d96-4cde-8db5-87a0bc38bc6b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:01.567855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 498/1019] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:27:01 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:27:01.599733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb71a2-c0a8-4405-a83b-7178289eac9c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:01.645786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b85e329-b616-44b9-aba2-63e86d40a60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:01.646454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b85e329-b616-44b9-aba2-63e86d40a60e None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.646729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b85e329-b616-44b9-aba2-63e86d40a60e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.647029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b85e329-b616-44b9-aba2-63e86d40a60e None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.647694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b85e329-b616-44b9-aba2-63e86d40a60e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:01.649813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b85e329-b616-44b9-aba2-63e86d40a60e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:01.690986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 522/1020] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:27:01 2022] POST /identity/v3/projects => generated 311 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:01.893026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5262c3-c273-4d53-aa7a-6ed1e1f85db5 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.893214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5262c3-c273-4d53-aa7a-6ed1e1f85db5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.893493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5262c3-c273-4d53-aa7a-6ed1e1f85db5 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.894827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 499/1021] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:01.900951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:01.901244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:01.901522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:01.920003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:01.921196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:01.986835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:01.992102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 523/1022] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:02.270923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5262c3-c273-4d53-aa7a-6ed1e1f85db5 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:02.378689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5262c3-c273-4d53-aa7a-6ed1e1f85db5 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:02.412019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:02.412644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:02.412922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:02.413189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:02.423606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:02.428328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:02.429360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:27:02.430394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:02.438038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 500/1023] 10.222.0.20 () {56 vars in 1120 bytes} [Fri Sep 9 04:27:02 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:02.438187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:02.543284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc63d4d-a3a7-4f92-8398-07a871067006 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:02.577830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:02.578489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:02.578774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:02.579079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:02.579738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:02.580040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:02.581399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:02.594483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 524/1024] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:27:02 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:02.595708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:02.607068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:02.699116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f17e66-5f1a-4bc8-9222-f0191a0ab705 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:02.732727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:02.733353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:02.733633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:02.733908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:02.743904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:02.745350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:02.760695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:02.761669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:02.769134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:02.769134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:02.769134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:02.769134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 501/1025] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:27:02.769440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:02.832897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:02.872724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613fb4d-404d-4959-add6-5258b28e6ff0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:02.906454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:02.907081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:02.907359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:02.907626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:02.908259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:02.908557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:02.909899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:02.921744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 525/1026] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:02.927125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:02.986133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:03.027850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c40617f-91bb-4d5c-9c95-17bbdcb9abba None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:03.060948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:03.061568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.061844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.062110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.069443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:03.071449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:27:03.082315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:03.083368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:27:03.084766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:03.087294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 502/1027] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:27:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:03.089107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:03.188476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683a18c8-a141-4dae-ae28-cf6053876a38 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:03.210147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:03.222499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:03.223128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.223406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.223676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.224395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:03.224698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:03.226108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:03.251834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 526/1028] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:27:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:03.253612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:03.361056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24a5f55-c31e-40f1-b70e-e13e15c98e31 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:03.382295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:03.397095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:03.397703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.397978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.398251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.465563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:03.467052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:03.569784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 503/1029] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:03 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:03.572217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:03.673957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b70d82d-f607-4c2b-a04f-578aaf203067 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:03.707839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:03.708465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.708739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.709009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.719543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:03.720971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-91cae05a-232f-4fb2-b717-3c635bf7976f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:03.736368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:03.737448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:03.743607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:03.743607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:03.743607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:03.743607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 527/1030] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:03.745499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:03.747372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3033a0ad-fb67-4c63-ac9d-79bb5e210ae0 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.747652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3033a0ad-fb67-4c63-ac9d-79bb5e210ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.747926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3033a0ad-fb67-4c63-ac9d-79bb5e210ae0 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.749235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 528/1031] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:27:03.754228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.754515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.754794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.767919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:03.773441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:03.775186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:03.804711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:03.856633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 529/1032] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:03 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:27:03.857249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653f899-c526-43ce-80bb-865e9899c51c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:03.891520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:03.892127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.892408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.892682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.893320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:03.893626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:03.895026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb77d194-d70f-48c1-9a14-a39a69cb3498 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:03.911416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 504/1033] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:03.912678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:03.939101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e6449-61ec-4fda-8e2e-30df39d220b4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.939284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e6449-61ec-4fda-8e2e-30df39d220b4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.939535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e6449-61ec-4fda-8e2e-30df39d220b4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.940881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 505/1034] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:27:03.946236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:03.946522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:03.946800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:03.966863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:03.968074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:04.020463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b8f88-f20c-4a85-816b-65f4f0588d24 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.038378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 506/1035] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:03 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:27:04.057549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.058496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.058950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.059380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.066733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.068353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:27:04.079849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:04.080927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:27:04.082057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:04.084058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 530/1036] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:27:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:04.092097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.105398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.130009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.160139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.197914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c975579e-e7a1-42d0-9871-305fc76e495d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.232939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.233556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.233839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.234116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.234814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:04.235155 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.236603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.244403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69493360-0022-41a5-880b-59f4882a23c7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.248327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 507/1037] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:04.254888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.286808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.287608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.287905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.288220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] PATH_INFO: `/v3/services/image` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.288803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.290284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:04.300430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:04.301541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:04.302731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 531/1038] 10.222.0.20 () {56 vars in 1105 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/services/image => generated 86 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:27:04.308162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.332947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.364087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67424ab-1156-4ddf-991f-0c361e5728e9 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.393005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.399363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.399966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.400244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.400513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.401233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] RBAC: Adding query filter params (role.id=75781bdd4b9e45a49f9e77b94ad8b0e6, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=0dd190c922964d51af263442650b427d) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:04.401562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.403000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.408240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 508/1039] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/role_assignments?user.id=0dd190c922964d51af263442650b427d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=75781bdd4b9e45a49f9e77b94ad8b0e6 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:04.408663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae2056-4005-446f-870e-de27a5e168ed None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.409001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.456620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.457147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.457428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.457703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.458337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] RBAC: Adding query filter params (name=image) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:04.458645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.460078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.465981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 532/1040] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/services?name=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:04.472543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.530831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae433bf-ac5d-4470-bc5b-df5b7e241c4e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.572644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.573283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.573556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.573826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.574993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.576482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.578361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 509/1041] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:04.584092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.587743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.591414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c47ec38-ab02-48b6-94f8-1fd3e426cef1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.629357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.629969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.630255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.630532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.631194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] RBAC: Adding query filter params (type=image) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:04.631500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.632866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.638568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 533/1042] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/services?type=image => generated 350 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:04.644319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.662566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.694119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a7db70-6754-4d96-bd59-9435c586ba3b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.736933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.737471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.738013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.738501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.739671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.741510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.750989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d83a1-08fd-46fe-a59a-274283b2c8aa None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.775604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 510/1043] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:27:04 2022] POST /identity/v3/users => generated 285 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:27:04.798072 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.798750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.799115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.799398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.800124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] RBAC: Adding query filter params (interface=public, service_id=92c9f91cd1884de88ca3c6bbc0aebb8e, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:04.800476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.801898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b445a8-1d08-4994-8df1-e1971f32e125 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.813046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 534/1044] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/endpoints?service_id=92c9f91cd1884de88ca3c6bbc0aebb8e&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:04.814329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:04.883070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:04.932995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2172cf37-a919-4f21-b74d-4229ba3b6c54 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:04.971052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc95c8-f62e-4259-b756-2e1e3c19d3cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:04.971667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc95c8-f62e-4259-b756-2e1e3c19d3cc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:04.971946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc95c8-f62e-4259-b756-2e1e3c19d3cc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:04.972214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc95c8-f62e-4259-b756-2e1e3c19d3cc None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:04.972858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc95c8-f62e-4259-b756-2e1e3c19d3cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:04.974190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc95c8-f62e-4259-b756-2e1e3c19d3cc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:04.981254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 511/1045] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:27:04 2022] GET /identity/v3/services => generated 1957 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:27:06.545885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009eb858-4176-4705-ade7-0043ed165003 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:06.546392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009eb858-4176-4705-ade7-0043ed165003 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:06.546392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009eb858-4176-4705-ade7-0043ed165003 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:06.547839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 535/1046] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:06.552822 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4cb1f-852c-4ed2-b9fa-e3d0cf4843d4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:06.553246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4cb1f-852c-4ed2-b9fa-e3d0cf4843d4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:06.553409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4cb1f-852c-4ed2-b9fa-e3d0cf4843d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:06.573810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-afc4cb1f-852c-4ed2-b9fa-e3d0cf4843d4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:06.575024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-afc4cb1f-852c-4ed2-b9fa-e3d0cf4843d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:06.647265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 512/1047] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:06.909434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009eb858-4176-4705-ade7-0043ed165003 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:06.994719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.019860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009eb858-4176-4705-ade7-0043ed165003 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.026602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd43336-ddbe-4e27-befa-4c8117163087 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.027024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd43336-ddbe-4e27-befa-4c8117163087 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.027198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd43336-ddbe-4e27-befa-4c8117163087 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.028787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 513/1048] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:07.034483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4eb3a-a526-42be-bafc-c56989bff04e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.034772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4eb3a-a526-42be-bafc-c56989bff04e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.035075 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4eb3a-a526-42be-bafc-c56989bff04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.053384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-dce4eb3a-a526-42be-bafc-c56989bff04e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:07.054580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dce4eb3a-a526-42be-bafc-c56989bff04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:07.061798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.062855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.063176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.063469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.075940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:07.080910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:07.081860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:07.083376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 536/1049] 10.222.0.20 () {56 vars in 1120 bytes} [Fri Sep 9 04:27:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:07.088345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.106989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.129545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 514/1050] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:07 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:27:07.140905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c1ab2b-0827-4132-9460-ef5a076f7525 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.141281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c1ab2b-0827-4132-9460-ef5a076f7525 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.141456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c1ab2b-0827-4132-9460-ef5a076f7525 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.142912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 515/1051] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:27:07.147531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.147814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.148090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.167037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:07.167856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:07.200950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d6d7e6-26c5-4d22-aaef-373039c251f5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.240700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 516/1052] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:07 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:27:07.243791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.244409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.244690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.244966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.245635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:07.245944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.247343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.252997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.256717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 537/1053] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:07.262255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.360804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97683d8d-386f-4add-8e2e-4ab5db7112c2 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.373543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.395861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.396490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.396762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.397027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.404860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.409261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.411588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:07.429407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:07.430418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:07.431362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:07.431362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:07.431362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:07.431540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 517/1054] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:07.437000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.517119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cecef9-8df6-463e-857b-14f8bebb7a19 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.543808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518caac-5e31-4cc0-adce-9023b158fa84 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.551963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.552598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.552881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.553156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.564091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:07.568931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:07.569856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:07.571102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 538/1055] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:27:07.572201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.585673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.586812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.587118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.587391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.588032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:07.588339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.589757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.607176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 518/1056] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:07.607176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.626284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.636122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.677173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a85a7fc-67e5-4e7e-8018-c5fffec90966 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.700186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab703fca-0fad-42c5-8072-4fa4fe419d58 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.710837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.712046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.712327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.712600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] PATH_INFO: `/v3/services/image` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.713177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.714533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:07.723148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:07.724253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:07.725077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 539/1057] 10.222.0.20 () {56 vars in 1105 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/services/image => generated 86 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:07.725873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.746787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.747429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.747709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.747980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.748648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:07.748950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.750314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.759040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 519/1058] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:07.759816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.828199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff2d486-1d28-4ff1-92b2-4762981c8982 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.842095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.852016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:07.856955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb869ad-117e-4b3a-ad7a-22a8183a4603 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.863317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.863927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.864207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.864478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.871805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.873240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:27:07.885050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:07.885989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 540/1059] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:07.887219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.893375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:07.893989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:07.894272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:07.894546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:07.895220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] RBAC: Adding query filter params (name=image) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:07.895527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:07.896920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:07.902956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 520/1060] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/services?name=image => generated 122 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:07.903093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:07.991926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f53a464-607a-403d-a5df-d37e5ae2bb32 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:07.997936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f9cdb8-138a-4518-bd7a-97ca92d8a15f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.025465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.026090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.026369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.026641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] PATH_INFO: `/v3/users/7a702610b90e49b08fb1908d5ccf72e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.030392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.031149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.031442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.031714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.032431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:08.032738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.034198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.039793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] RBAC: Authorizing `identity:get_user(user_id=7a702610b90e49b08fb1908d5ccf72e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.041210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.042820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 541/1061] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/users/7a702610b90e49b08fb1908d5ccf72e2 => generated 285 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:08.043617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.051476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 521/1062] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:27:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:08.052269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.063759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.091742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.149070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d3d582-b2ae-4d20-9ebb-b815beab21f6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.154725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d717cbd7-4739-4147-b6eb-e443eb9de17c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.183690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.184301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.184600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.184855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.185485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] RBAC: Adding query filter params (type=image) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:08.185791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.187202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.189109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.189703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.189981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.190250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.192061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 542/1063] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:08.192835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.256768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.258769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.268519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.292027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b27b47-1bd6-4fd0-87d9-55d5efb0691d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.299444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.327283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.327906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.328185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.328457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.338400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.339876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:08.354393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:08.355531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:08.356533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:08.356533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:08.356533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:08.356533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 543/1064] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:08.356757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 522/1065] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:08.356846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.364962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.454595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad00db5-dc97-45a2-b3b8-8600ac34ac10 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.457522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fd0790-0923-4cde-9568-1621f104729c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.468224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.488327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.488922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.489197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.489467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.490124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.491505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.492717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.493314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.493591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.493861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.494494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:08.494793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.496178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.507273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 523/1066] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:08.508615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.519154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 544/1067] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:27:08 2022] POST /identity/v3/endpoints => generated 327 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:08.520531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.541905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.621729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b3a807-0d77-4df6-b843-4b73384d1fe3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.654190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea49d50-f76d-41d7-95c4-687233741f6a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.658247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.658910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.659208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.659479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.660643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] RBAC: Authorizing `identity:get_project(project_id=010a1465473047d58f45da77236419e2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.662106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.669580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 524/1068] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/projects/010a1465473047d58f45da77236419e2 => generated 311 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:08.670605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.701050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.701681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.701958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.702229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.710511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.712375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.713988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:27:08.727638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:08.728597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:08.729519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 545/1069] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:08.734451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:08.793983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8729e3-c5ed-441e-b068-6d17aa709a38 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.818466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:08.830340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.830989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.831291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.831569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.832301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=010a1465473047d58f45da77236419e2, user.id=7a702610b90e49b08fb1908d5ccf72e2) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:08.832611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.834045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.846956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 525/1070] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/role_assignments?user.id=7a702610b90e49b08fb1908d5ccf72e2&scope.project.id=010a1465473047d58f45da77236419e2&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:08.851676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3d0a10-968f-44b7-b5d5-99d86c69dff6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:08.890911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:08.891556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:08.891835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:08.892113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:08.892820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:08.893122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:08.894587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a0a436-631a-45be-9330-b132786ccf04 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:08.908572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 546/1071] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:27:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:08.914503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:09.024254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:09.027196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40ac4b-5fbe-4f13-b69f-64ac1d6e413f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:09.062473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c7ebb1-a4f2-463e-9d2b-f3d8796f1a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:09.063094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c7ebb1-a4f2-463e-9d2b-f3d8796f1a63 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:09.063518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c7ebb1-a4f2-463e-9d2b-f3d8796f1a63 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:09.064021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c7ebb1-a4f2-463e-9d2b-f3d8796f1a63 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/0dd190c922964d51af263442650b427d/roles/75781bdd4b9e45a49f9e77b94ad8b0e6` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:09.067702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c7ebb1-a4f2-463e-9d2b-f3d8796f1a63 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=0dd190c922964d51af263442650b427d, role_id=75781bdd4b9e45a49f9e77b94ad8b0e6)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:09.069739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c7ebb1-a4f2-463e-9d2b-f3d8796f1a63 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:09.090969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 526/1072] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:27:08 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/0dd190c922964d51af263442650b427d/roles/75781bdd4b9e45a49f9e77b94ad8b0e6 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:27:10.703791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24df8dec-41a8-4097-88ce-c299ce4e4d85 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:10.704320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24df8dec-41a8-4097-88ce-c299ce4e4d85 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:10.704320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24df8dec-41a8-4097-88ce-c299ce4e4d85 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:10.705712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 547/1073] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:10.710787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15c37b-58c7-4249-8618-e3c546abcb02 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:10.711769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15c37b-58c7-4249-8618-e3c546abcb02 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:10.711933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15c37b-58c7-4249-8618-e3c546abcb02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:10.730241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bf15c37b-58c7-4249-8618-e3c546abcb02 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:10.731467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bf15c37b-58c7-4249-8618-e3c546abcb02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:10.800934 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 527/1074] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:10.999219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671f2db-afd2-42d8-be72-fe0d63416fdb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:10.999555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671f2db-afd2-42d8-be72-fe0d63416fdb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:10.999731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671f2db-afd2-42d8-be72-fe0d63416fdb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.001163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 548/1075] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:11.006076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f214297-2a88-47c9-878e-3c0a8b281362 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.006371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f214297-2a88-47c9-878e-3c0a8b281362 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.006647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f214297-2a88-47c9-878e-3c0a8b281362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.025223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5f214297-2a88-47c9-878e-3c0a8b281362 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:11.026399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f214297-2a88-47c9-878e-3c0a8b281362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:11.065235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3671f2db-afd2-42d8-be72-fe0d63416fdb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.094523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 528/1076] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:11.171996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3671f2db-afd2-42d8-be72-fe0d63416fdb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.200967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.207811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.208419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.208696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.208964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.262923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f082c29-c25b-483b-bc92-6c9334ff0cb0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.263262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f082c29-c25b-483b-bc92-6c9334ff0cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.263541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f082c29-c25b-483b-bc92-6c9334ff0cb0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.265014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 529/1077] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:27:11.269739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.270031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.270313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.287582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:11.289056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:11.289366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:11.290557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:11.307195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.365220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 530/1078] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 09 04:27:11.366314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.395105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 549/1079] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 09 04:27:11.399400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.429549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.474702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43027aa-2014-49eb-8256-24b80d8a0c66 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.501953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456508a4-8784-43ac-bea3-76cdf1bcd2b7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.513976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.514543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.514819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.515126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.525622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:11.530689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:11.531643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:11.532696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 531/1080] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:11.537656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.541293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.541918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.542835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.542835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.550974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.553538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:11.555589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:11.572523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:11.573476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:11.574354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:11.574354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:11.574354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:11.574354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 550/1081] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:11.579469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.648672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98808853-2756-43f0-9c56-0bc656507477 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.668460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.683443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.684044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.684320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.684588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.685240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:11.685542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:11.686906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:11.687310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeadaa3-4d4d-4534-b676-cffbb43655eb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.694977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 532/1082] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:11.695758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.723900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.724622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.724906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.725185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.725823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:11.726131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:11.728057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:11.739409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 551/1083] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:11.740201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.775892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.813140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ed5a5-5468-44e9-bd01-07be076990c4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.849924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.850566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.850845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.851147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.859976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f0735-e786-49f7-934e-9b99c48c0863 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:11.863246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:11.868756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:11.869775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 09 04:27:11.870621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:11.875317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 533/1084] 10.222.0.20 () {56 vars in 1120 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:11.876150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.902132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:11.902774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:11.903092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:11.903368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] PATH_INFO: `/v3/users/7a702610b90e49b08fb1908d5ccf72e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:11.904520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] RBAC: Authorizing `identity:get_user(user_id=7a702610b90e49b08fb1908d5ccf72e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:11.905952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:11.907565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 552/1085] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/users/7a702610b90e49b08fb1908d5ccf72e2 => generated 285 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:11.913653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:11.926378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:11.999035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a4f348-41f0-4838-93d9-92395fc254c7 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.030675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:12.034803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776d03d0-1438-43a5-a7e5-463d42d8f866 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.040327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.040939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.041214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.041486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.042149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:12.042453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.044504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.053683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 534/1086] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:12.060249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:12.070297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.071039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.071322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.071592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.072707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] RBAC: Authorizing `identity:get_project(project_id=010a1465473047d58f45da77236419e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.074163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.075814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 553/1087] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:11 2022] GET /identity/v3/projects/010a1465473047d58f45da77236419e2 => generated 311 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:12.081543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:12.156751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:12.160817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeeafcb-deb2-43d7-8f37-4583e307b640 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.178871 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cc13a-39f0-4838-ad60-d74fbbffc670 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.194610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.195759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.196044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.196311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.206004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.207481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.213503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.214154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.214440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.214725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2/users/7a702610b90e49b08fb1908d5ccf72e2/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.216791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] RBAC: Authorizing `identity:create_grant(project_id=010a1465473047d58f45da77236419e2, user_id=7a702610b90e49b08fb1908d5ccf72e2, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.218200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:12.222317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:12.223535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:12.224768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:12.224768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:12.224768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:12.224768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 535/1088] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:27:12.229579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:12.232095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 554/1089] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:27:12 2022] PUT /identity/v3/projects/010a1465473047d58f45da77236419e2/users/7a702610b90e49b08fb1908d5ccf72e2/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:27:12.283326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:12.340443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56139dc-5646-471d-9091-1a7ce25067f2 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.376397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.377020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.377301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.377575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.378233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:12.378539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.379976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.393602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 536/1090] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:27:12.398368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:12.498309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8e77b-c0a7-4eae-8e7a-1fb7cb81277f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.532187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.532814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.533093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.533417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.540867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.542288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.544047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:27:12.555436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:12.556619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:12.558204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 555/1091] 10.222.0.20 () {56 vars in 1093 bytes} [Fri Sep 9 04:27:12 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:12.562531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:12.659816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdce93d-a8fd-4322-ab63-a7354798bb4b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.667580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:12.694379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.695011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.695328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.695602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.696326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:12.696628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.698045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:12.717897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 537/1092] 10.222.0.20 () {56 vars in 1133 bytes} [Fri Sep 9 04:27:12 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:12.723878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:12.822511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac0ac4d-039a-4316-84b8-7774c5aad622 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:12.854560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:12.855216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:12.855499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:12.855773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:12.918993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:12.923305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:12.924767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:13.021613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 556/1093] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:13.028049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:13.126039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4b1cb-130a-4d9e-954c-496caee46e08 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:13.166413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:13.167665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:13.167665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:13.167881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:13.177624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:13.179091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:13.194302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:13.196242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:27:13.197469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:13.197469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:27:13.197469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:13.197469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 538/1094] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:13.201735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:13.295559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:13.299671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86feca0f-bb31-4d99-a47f-4eaebc051d39 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:13.334635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:13.335304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:13.335582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:13.335851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:13.336483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:13.336782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:13.338126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:13.349093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 557/1095] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:27:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:13.354692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:13.373166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:13.453765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25939e54-b847-4803-9e42-9286638bca74 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:13.486366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:13.487063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:13.487373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:13.487668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:13.494702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:13.496180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:27:13.507310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:13.508794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:27:13.509673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:13.511215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 539/1096] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:27:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:27:13.514690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:13.612057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ed918-8f5f-4df2-8f07-b17f72adc371 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:13.645182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:13.645773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:13.646049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:13.646318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:13.647037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:13.647340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:13.648772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca43c26-3c26-4d32-85ed-be69eb1ef192 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:13.660176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 558/1097] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:27:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:13.666924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:13.744199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:13.767464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be4f7b5-e477-4aef-98bc-3570f5c75d4d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:13.799902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:13.800495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:13.800769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:13.801038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:13.801752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] RBAC: Adding query filter params (role.id=75781bdd4b9e45a49f9e77b94ad8b0e6, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=0dd190c922964d51af263442650b427d) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:13.802056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:13.803517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94edfaa7-c224-40ee-b46c-1142b7f25993 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:13.809958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 540/1098] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:13 2022] GET /identity/v3/role_assignments?user.id=0dd190c922964d51af263442650b427d&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=75781bdd4b9e45a49f9e77b94ad8b0e6 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:14.372349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488056f1-6c94-4094-acf1-f25fa0bdc44c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:14.372893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488056f1-6c94-4094-acf1-f25fa0bdc44c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:14.372995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488056f1-6c94-4094-acf1-f25fa0bdc44c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:14.374336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 559/1099] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:14.379668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:14.380118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:14.380224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:14.398922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:14.400157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:14.467535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 541/1100] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:14.735701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488056f1-6c94-4094-acf1-f25fa0bdc44c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:14.754995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:14.835492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488056f1-6c94-4094-acf1-f25fa0bdc44c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:14.869202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:14.869821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:14.870099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:14.870370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:14.880281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:14.885018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:14.886121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:14.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 560/1101] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:14.892109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:14.958434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:14.988899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d77350d-8e6d-4e9e-8e40-1d1be649b800 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:15.020770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:15.021364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:15.021641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:15.021913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:15.022571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:15.022874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:15.024815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:15.032503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 542/1102] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:15.038231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:15.114822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:15.135662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549c8172-6984-486b-940b-fc084002c9a2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:15.168562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:15.169164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:15.169439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:15.169710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] PATH_INFO: `/v3/users/7a702610b90e49b08fb1908d5ccf72e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:15.170805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] RBAC: Authorizing `identity:get_user(user_id=7a702610b90e49b08fb1908d5ccf72e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:15.172244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:15.173801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 561/1103] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:15 2022] GET /identity/v3/users/7a702610b90e49b08fb1908d5ccf72e2 => generated 285 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:15.179434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:15.279426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebe9fbd-baac-4979-b7ff-22b20c45c852 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:15.306470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:15.312158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:15.312757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:15.313036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:15.313306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:15.377835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:15.379316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:15.473655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 543/1104] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:15.480103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:15.576456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d888da-ec35-4414-93d0-a3a9ae2b41b2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:15.609244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1897024-e09e-405c-92b4-2b3147672b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:15.609854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897024-e09e-405c-92b4-2b3147672b00 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:15.610130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897024-e09e-405c-92b4-2b3147672b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:15.610397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897024-e09e-405c-92b4-2b3147672b00 None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:15.611507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897024-e09e-405c-92b4-2b3147672b00 None admin] RBAC: Authorizing `identity:get_project(project_id=010a1465473047d58f45da77236419e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:15.612920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897024-e09e-405c-92b4-2b3147672b00 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:15.614515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 562/1105] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:15 2022] GET /identity/v3/projects/010a1465473047d58f45da77236419e2 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:15.620685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:15.650148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:15.715854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60fd14-80c5-4f6f-8312-bbb77558625f None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:15.749182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:15.749771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:15.750047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:15.750314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:15.751063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=010a1465473047d58f45da77236419e2, user.id=7a702610b90e49b08fb1908d5ccf72e2) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:15.751370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:15.752781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a977d7f-a082-49c8-8b68-a624ad4678b2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:15.758309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 544/1106] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:15 2022] GET /identity/v3/role_assignments?user.id=7a702610b90e49b08fb1908d5ccf72e2&scope.project.id=010a1465473047d58f45da77236419e2&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:17.906174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363a4f07-8ae8-4b04-99a2-8c9e9f6c2a69 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:17.906719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363a4f07-8ae8-4b04-99a2-8c9e9f6c2a69 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:17.906719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363a4f07-8ae8-4b04-99a2-8c9e9f6c2a69 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:17.908073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 563/1107] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:17.913252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:17.913677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:17.913879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:17.932183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:17.933369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:17.997327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 545/1108] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:18.263600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363a4f07-8ae8-4b04-99a2-8c9e9f6c2a69 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:18.279844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:18.362907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363a4f07-8ae8-4b04-99a2-8c9e9f6c2a69 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:18.395307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0b153-29ad-40c9-b34a-51f8045b4ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:18.395917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0b153-29ad-40c9-b34a-51f8045b4ead None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:18.396197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0b153-29ad-40c9-b34a-51f8045b4ead None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:18.396469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0b153-29ad-40c9-b34a-51f8045b4ead None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:18.397574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0b153-29ad-40c9-b34a-51f8045b4ead None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:18.399078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0b153-29ad-40c9-b34a-51f8045b4ead None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:18.400607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 564/1109] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:18.406345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:18.433391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:18.502560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bf5802-d92f-408a-8c72-e7425b426f7c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:18.534992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab09e2e3-b1cd-4669-b1b9-bcd6d6a3c817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:18.535539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab09e2e3-b1cd-4669-b1b9-bcd6d6a3c817 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:18.535843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab09e2e3-b1cd-4669-b1b9-bcd6d6a3c817 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:18.536119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab09e2e3-b1cd-4669-b1b9-bcd6d6a3c817 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:18.536770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab09e2e3-b1cd-4669-b1b9-bcd6d6a3c817 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:18.538193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab09e2e3-b1cd-4669-b1b9-bcd6d6a3c817 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:18.558655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 546/1110] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:27:18 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:20.717771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce94aa-28cb-4679-8f53-550f1eef8b47 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:20.718324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce94aa-28cb-4679-8f53-550f1eef8b47 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:20.718324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce94aa-28cb-4679-8f53-550f1eef8b47 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:20.719676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 565/1111] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:20.724847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:20.725293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:20.725401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:20.742961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:20.744135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:20.808379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 547/1112] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:21.070006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce94aa-28cb-4679-8f53-550f1eef8b47 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:21.149420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:21.168420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce94aa-28cb-4679-8f53-550f1eef8b47 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:21.200900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:21.201502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:21.201778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:21.202048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] PATH_INFO: `/v3/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:21.203432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] RBAC: Authorizing `identity:get_role(role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:21.204777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:21.206053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 566/1113] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:21 2022] GET /identity/v3/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:21.212339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:21.226014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:21.310210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9912f17-88bd-4762-9646-1372e378e900 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:21.344013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:21.344573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:21.344852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:21.345125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] PATH_INFO: `/v3/users/fd251fc3d54847239faa1273453d2c0f` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:21.357711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] RBAC: Authorizing `identity:get_user(user_id=fd251fc3d54847239faa1273453d2c0f)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:21.359176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:21.360741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 548/1114] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:21 2022] GET /identity/v3/users/fd251fc3d54847239faa1273453d2c0f => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:21.366591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:21.463235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670af059-a6d0-4d91-9a4e-3af7f6d1efb2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:21.496417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:21.497013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:21.497293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:21.497564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:21.516275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:21.563815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:21.565244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:21.658546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 567/1115] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 09 04:27:21.664501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:21.730849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:21.761289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a2d11d-da42-4198-9485-43c1937562d4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:21.793535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f4260a-39f6-4d9c-b311-a73e4917a169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:21.794156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f4260a-39f6-4d9c-b311-a73e4917a169 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:21.794435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f4260a-39f6-4d9c-b311-a73e4917a169 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:21.794707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f4260a-39f6-4d9c-b311-a73e4917a169 None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:21.795836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f4260a-39f6-4d9c-b311-a73e4917a169 None admin] RBAC: Authorizing `identity:get_project(project_id=010a1465473047d58f45da77236419e2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:21.797265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f4260a-39f6-4d9c-b311-a73e4917a169 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:21.798873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 549/1116] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:21 2022] GET /identity/v3/projects/010a1465473047d58f45da77236419e2 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:21.805525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:21.853938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:21.902864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a83084-e68b-4c53-96ba-c7d14aa38d49 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:21.936771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:21.937311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:21.937592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:21.937865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:21.938580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] RBAC: Adding query filter params (role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=010a1465473047d58f45da77236419e2, user.id=fd251fc3d54847239faa1273453d2c0f) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:21.938886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:21.940322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cce2dea-aa4b-42b5-83f6-f465fc9c548f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:21.945533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 568/1117] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:21 2022] GET /identity/v3/role_assignments?user.id=fd251fc3d54847239faa1273453d2c0f&scope.project.id=010a1465473047d58f45da77236419e2&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:24.033121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c491fbe-3487-4e56-a230-22b4f84fb3b4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:24.033668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c491fbe-3487-4e56-a230-22b4f84fb3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:24.033668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c491fbe-3487-4e56-a230-22b4f84fb3b4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:24.035019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 550/1118] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:24.040027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:24.040435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:24.040572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:24.059001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:24.060171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:24.123100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 569/1119] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:24.390723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c491fbe-3487-4e56-a230-22b4f84fb3b4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:24.489394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c491fbe-3487-4e56-a230-22b4f84fb3b4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:24.522028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:24.522644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:24.523568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:24.523843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] PATH_INFO: `/v3/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:24.525218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] RBAC: Authorizing `identity:get_role(role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:24.526584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:24.527925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 551/1120] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:24 2022] GET /identity/v3/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:24.533766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:24.578356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:24.629999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690281bc-9ad7-412e-ad0f-689a44f0e2df None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:24.662463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:24.663099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:24.663378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:24.663684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] PATH_INFO: `/v3/users/fd251fc3d54847239faa1273453d2c0f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:24.664783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] RBAC: Authorizing `identity:get_user(user_id=fd251fc3d54847239faa1273453d2c0f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:24.666184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:24.667778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 570/1121] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:24 2022] GET /identity/v3/users/fd251fc3d54847239faa1273453d2c0f => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:24.673336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:24.682675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:24.771018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69f974-a092-42c4-979c-0c1c9f251d52 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:24.803271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6883dd-cead-4f82-9946-a8463334dcea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:24.803909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6883dd-cead-4f82-9946-a8463334dcea None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:24.804693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6883dd-cead-4f82-9946-a8463334dcea None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:24.804963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6883dd-cead-4f82-9946-a8463334dcea None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:24.806085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6883dd-cead-4f82-9946-a8463334dcea None admin] RBAC: Authorizing `identity:get_project(project_id=010a1465473047d58f45da77236419e2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:24.807817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6883dd-cead-4f82-9946-a8463334dcea None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:24.809440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 552/1122] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:24 2022] GET /identity/v3/projects/010a1465473047d58f45da77236419e2 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:24.815465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:24.911581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075e5d3-f10a-4451-8739-19d3df7770b7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:24.924899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:24.946780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454854f-28d4-4f75-a23a-06a6398b00f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:24.947460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454854f-28d4-4f75-a23a-06a6398b00f9 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:24.947743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454854f-28d4-4f75-a23a-06a6398b00f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:24.948017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454854f-28d4-4f75-a23a-06a6398b00f9 None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2/users/fd251fc3d54847239faa1273453d2c0f/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:24.950157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454854f-28d4-4f75-a23a-06a6398b00f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=010a1465473047d58f45da77236419e2, user_id=fd251fc3d54847239faa1273453d2c0f, role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:24.951624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454854f-28d4-4f75-a23a-06a6398b00f9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:24.963705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 571/1123] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:27:24 2022] PUT /identity/v3/projects/010a1465473047d58f45da77236419e2/users/fd251fc3d54847239faa1273453d2c0f/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:27:27.058906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b157b82-9df5-4200-b1a9-b89e4039086d None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:27.059497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b157b82-9df5-4200-b1a9-b89e4039086d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:27.059497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b157b82-9df5-4200-b1a9-b89e4039086d None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:27.060858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 553/1124] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:27.065977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:27.066411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:27.066513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:27.084961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:27.086630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:27.150039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 572/1125] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:27.414681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b157b82-9df5-4200-b1a9-b89e4039086d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:27.488057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:27.513189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b157b82-9df5-4200-b1a9-b89e4039086d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:27.545630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:27.546250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:27.546529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:27.546802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] PATH_INFO: `/v3/roles/452a3efe1ad847f3b9e17732af3e6f33` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:27.548199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] RBAC: Authorizing `identity:get_role(role_id=452a3efe1ad847f3b9e17732af3e6f33)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:27.549561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:27.550850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 554/1126] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:27 2022] GET /identity/v3/roles/452a3efe1ad847f3b9e17732af3e6f33 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:27.557243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:27.652231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0db6ea-5de6-400b-9de9-0cb7cd2231eb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:27.653724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:27.687273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:27.687828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:27.688105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:27.688377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] PATH_INFO: `/v3/users/fd251fc3d54847239faa1273453d2c0f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:27.689484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] RBAC: Authorizing `identity:get_user(user_id=fd251fc3d54847239faa1273453d2c0f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:27.690902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:27.692471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 573/1127] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:27 2022] GET /identity/v3/users/fd251fc3d54847239faa1273453d2c0f => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:27.698520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:27.793431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dd5fd7-6350-41ac-9f0f-91b9bed4d3fc None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:27.826785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:27.827401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:27.827680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:27.827951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:27.839619 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:27.892648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:27.894077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:27.988551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 555/1128] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:27.994359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:28.090282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e1ec10-8b30-421d-bf34-ba27b3f139be None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:28.122316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d823dd2a-9c1b-49db-a149-c2a9bab80f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:28.122955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d823dd2a-9c1b-49db-a149-c2a9bab80f36 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:28.123237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d823dd2a-9c1b-49db-a149-c2a9bab80f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:28.123509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d823dd2a-9c1b-49db-a149-c2a9bab80f36 None admin] PATH_INFO: `/v3/projects/010a1465473047d58f45da77236419e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:28.124615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d823dd2a-9c1b-49db-a149-c2a9bab80f36 None admin] RBAC: Authorizing `identity:get_project(project_id=010a1465473047d58f45da77236419e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:28.126033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d823dd2a-9c1b-49db-a149-c2a9bab80f36 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:28.127661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 574/1129] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:27 2022] GET /identity/v3/projects/010a1465473047d58f45da77236419e2 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:28.134157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:28.178237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:28.233108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14ea8e1-e1e8-4bca-b7a9-e5ac65269f9b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:28.267302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:28.267900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:28.268178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:28.268449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:28.269172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] RBAC: Adding query filter params (role.id=452a3efe1ad847f3b9e17732af3e6f33, scope.project.id=010a1465473047d58f45da77236419e2, user.id=fd251fc3d54847239faa1273453d2c0f) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:28.269479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:28.270898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b6508-4689-4160-aff1-4c34059ab0fc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:28.277115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 556/1130] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:28 2022] GET /identity/v3/role_assignments?user.id=fd251fc3d54847239faa1273453d2c0f&scope.project.id=010a1465473047d58f45da77236419e2&role.id=452a3efe1ad847f3b9e17732af3e6f33 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:30.419207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11cd2e0-52ba-4e08-be0d-1c8b9b4f903b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:30.419752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11cd2e0-52ba-4e08-be0d-1c8b9b4f903b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:30.419752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11cd2e0-52ba-4e08-be0d-1c8b9b4f903b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:30.421076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 575/1131] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:30.426304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:30.426741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:30.426852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:30.445502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:30.446679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:30.510175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 557/1132] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:30.776773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11cd2e0-52ba-4e08-be0d-1c8b9b4f903b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:30.819313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:30.881681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11cd2e0-52ba-4e08-be0d-1c8b9b4f903b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:30.915070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465e5b6f-7fea-4b57-9060-12f4cc1fe8f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:30.915679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465e5b6f-7fea-4b57-9060-12f4cc1fe8f8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:30.915954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465e5b6f-7fea-4b57-9060-12f4cc1fe8f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:30.916225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465e5b6f-7fea-4b57-9060-12f4cc1fe8f8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:30.917321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465e5b6f-7fea-4b57-9060-12f4cc1fe8f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:30.918730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465e5b6f-7fea-4b57-9060-12f4cc1fe8f8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:30.920303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 576/1133] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:30.926143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:30.959439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:31.021572 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981d597f-420c-4259-94c8-3e58c59083a6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:31.053753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c476c0d-2d35-4975-893e-5f1035104e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:31.054838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c476c0d-2d35-4975-893e-5f1035104e22 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:31.055148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c476c0d-2d35-4975-893e-5f1035104e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:31.055420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c476c0d-2d35-4975-893e-5f1035104e22 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:31.056072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c476c0d-2d35-4975-893e-5f1035104e22 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:31.057770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c476c0d-2d35-4975-893e-5f1035104e22 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:31.082147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 558/1134] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:27:30 2022] POST /identity/v3/projects => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:33.232343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb5515a-065f-4b33-b5bf-cc99bf7397db None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:33.232919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb5515a-065f-4b33-b5bf-cc99bf7397db None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:33.232919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb5515a-065f-4b33-b5bf-cc99bf7397db None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:33.234214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 577/1135] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:33.239508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:33.239955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:33.240056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:33.258480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:33.259689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:33.325073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 559/1136] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:33.591561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb5515a-065f-4b33-b5bf-cc99bf7397db None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:33.689489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb5515a-065f-4b33-b5bf-cc99bf7397db None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:33.696528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:33.722548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd4808b-a296-4c51-b9ec-743706cea0d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:33.723187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd4808b-a296-4c51-b9ec-743706cea0d9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:33.723469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd4808b-a296-4c51-b9ec-743706cea0d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:33.723743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd4808b-a296-4c51-b9ec-743706cea0d9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:33.724849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd4808b-a296-4c51-b9ec-743706cea0d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:33.726341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd4808b-a296-4c51-b9ec-743706cea0d9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:33.727902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 578/1137] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:27:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:33.733467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:33.751643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:33.830124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2bf273-4e2b-445b-8d09-cc1d509b171d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:33.861887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6d98f3-82c7-4f4f-80d7-47a036e82875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:33.862421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6d98f3-82c7-4f4f-80d7-47a036e82875 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:33.862700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6d98f3-82c7-4f4f-80d7-47a036e82875 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:33.863003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6d98f3-82c7-4f4f-80d7-47a036e82875 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:33.863754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6d98f3-82c7-4f4f-80d7-47a036e82875 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:33.865181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6d98f3-82c7-4f4f-80d7-47a036e82875 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:33.885480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 560/1138] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:27:33 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:36.029509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53e390-2fe4-4183-a1b1-4c8b33446590 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:36.030073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53e390-2fe4-4183-a1b1-4c8b33446590 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:36.030073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53e390-2fe4-4183-a1b1-4c8b33446590 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:36.031428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 579/1139] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:36.036615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:36.037045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:36.037168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:36.056280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:36.057700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:36.121158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 561/1140] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:36.387532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53e390-2fe4-4183-a1b1-4c8b33446590 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:36.489014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53e390-2fe4-4183-a1b1-4c8b33446590 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:36.521180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:36.521788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:36.522066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:36.522337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:36.532627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:36.537285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:36.538469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:36.539802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 580/1141] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:36.544265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:36.547301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:36.641770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc3b11-f19b-45a7-b69f-dbb5c13a64fa None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:36.673750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:36.674347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:36.674626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:36.674901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:36.675590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:36.675894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:36.677263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:36.685829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 562/1142] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:36.691647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:36.726315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:36.790714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f308d6-b9b4-4b54-8db1-7c347d1d2e98 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:36.824621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:36.825242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:36.825528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:36.825800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] PATH_INFO: `/v3/users/c2580191d3a947798bc83d976d1cc85f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:36.842227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] RBAC: Authorizing `identity:get_user(user_id=c2580191d3a947798bc83d976d1cc85f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:36.843704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:36.845318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 581/1143] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:36 2022] GET /identity/v3/users/c2580191d3a947798bc83d976d1cc85f => generated 286 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:36.851119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:36.914392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:36.949319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093e75f8-ed24-4af4-940f-b9c48e6894a0 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:36.981196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:36.981787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:36.982063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:36.982329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:37.046449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:37.047905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:37.108713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:37.143757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 563/1144] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:37.150270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:37.238904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:37.247663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdb94d0-7ec3-42f8-a14b-920a15a92698 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:37.281471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe4408-9da1-4016-9444-a7794db5cecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:37.282089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe4408-9da1-4016-9444-a7794db5cecb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:37.282369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe4408-9da1-4016-9444-a7794db5cecb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:37.282642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe4408-9da1-4016-9444-a7794db5cecb None admin] PATH_INFO: `/v3/projects/fd773b1b15ee4cad97121bec172aabf9` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:37.283828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe4408-9da1-4016-9444-a7794db5cecb None admin] RBAC: Authorizing `identity:get_project(project_id=fd773b1b15ee4cad97121bec172aabf9)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:37.285249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe4408-9da1-4016-9444-a7794db5cecb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:37.286859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 582/1145] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:37 2022] GET /identity/v3/projects/fd773b1b15ee4cad97121bec172aabf9 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:37.293127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:37.389292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a7f094-24b0-4248-8198-1e55c8809c6c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:37.421979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:37.422567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:37.422843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:37.423148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:37.423861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=fd773b1b15ee4cad97121bec172aabf9, user.id=c2580191d3a947798bc83d976d1cc85f) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:37.424160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:37.425565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4619a4-e73a-4e0a-ae41-b1a638f4099b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:37.431552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 564/1146] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:37 2022] GET /identity/v3/role_assignments?user.id=c2580191d3a947798bc83d976d1cc85f&scope.project.id=fd773b1b15ee4cad97121bec172aabf9&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:39.543157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba8286c-49a6-4125-984b-f42c43e7a733 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:39.543705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba8286c-49a6-4125-984b-f42c43e7a733 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:39.543705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba8286c-49a6-4125-984b-f42c43e7a733 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:39.544963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 583/1147] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:39.550022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:39.550423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:39.550561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:39.569740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:39.570920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:39.593832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:39.634350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 565/1148] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:39.901355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba8286c-49a6-4125-984b-f42c43e7a733 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:39.999495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba8286c-49a6-4125-984b-f42c43e7a733 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:40.032388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:40.032998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:40.033278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:40.033549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:40.043287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:40.048010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:40.049077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:40.049968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 584/1149] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:40.054911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:40.152061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78184eb-5c71-4127-82e1-8f9cd190345e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:40.184701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:40.185302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:40.185580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:40.185851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:40.186508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:40.186810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:40.188203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:40.196482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 566/1150] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:40.202382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:40.224604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:40.299839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dbfd4f-767f-412d-9988-531f79759696 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:40.332955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:40.333569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:40.333846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:40.334116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] PATH_INFO: `/v3/users/c2580191d3a947798bc83d976d1cc85f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:40.335261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] RBAC: Authorizing `identity:get_user(user_id=c2580191d3a947798bc83d976d1cc85f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:40.336679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:40.338240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 585/1151] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:40 2022] GET /identity/v3/users/c2580191d3a947798bc83d976d1cc85f => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:40.344092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:40.368667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:40.440922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751d95c8-a938-44d7-8ba0-a8abb3345ca2 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:40.472670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b72c83b-8554-46d3-92e1-b2c3499a89ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:40.473279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b72c83b-8554-46d3-92e1-b2c3499a89ef None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:40.473554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b72c83b-8554-46d3-92e1-b2c3499a89ef None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:40.473824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b72c83b-8554-46d3-92e1-b2c3499a89ef None admin] PATH_INFO: `/v3/projects/fd773b1b15ee4cad97121bec172aabf9` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:40.474910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b72c83b-8554-46d3-92e1-b2c3499a89ef None admin] RBAC: Authorizing `identity:get_project(project_id=fd773b1b15ee4cad97121bec172aabf9)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:40.476397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b72c83b-8554-46d3-92e1-b2c3499a89ef None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:40.477983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 567/1152] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:40 2022] GET /identity/v3/projects/fd773b1b15ee4cad97121bec172aabf9 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:40.483845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:40.574115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:40.585964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31b3494-ee1a-4ea2-b9fc-145dfb010479 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:40.618341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a79cc94-bf54-4e7c-be91-f5495cf00f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:40.619001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a79cc94-bf54-4e7c-be91-f5495cf00f8e None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:40.619281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a79cc94-bf54-4e7c-be91-f5495cf00f8e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:40.619552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a79cc94-bf54-4e7c-be91-f5495cf00f8e None admin] PATH_INFO: `/v3/projects/fd773b1b15ee4cad97121bec172aabf9/users/c2580191d3a947798bc83d976d1cc85f/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:40.621401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a79cc94-bf54-4e7c-be91-f5495cf00f8e None admin] RBAC: Authorizing `identity:create_grant(project_id=fd773b1b15ee4cad97121bec172aabf9, user_id=c2580191d3a947798bc83d976d1cc85f, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:40.622812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a79cc94-bf54-4e7c-be91-f5495cf00f8e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:40.635701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 586/1153] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:27:40 2022] PUT /identity/v3/projects/fd773b1b15ee4cad97121bec172aabf9/users/c2580191d3a947798bc83d976d1cc85f/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:27:42.731656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b46368-50c0-436c-a893-f1590f0c2b87 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:42.732191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b46368-50c0-436c-a893-f1590f0c2b87 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:42.732191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b46368-50c0-436c-a893-f1590f0c2b87 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:42.733522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 568/1154] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:42.738612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:42.739082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:42.739187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:42.757575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:42.758754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:42.823326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 587/1155] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:43.090544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b46368-50c0-436c-a893-f1590f0c2b87 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:43.179775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:43.190690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b46368-50c0-436c-a893-f1590f0c2b87 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:43.222630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:43.223267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:43.223544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:43.223814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:43.234084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:43.238827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:43.239958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:43.241647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 569/1156] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:43.246421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:43.321286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:43.343805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb41a45-0ee0-410d-a177-0bda83899508 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:43.376415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:43.377016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:43.377293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:43.377566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:43.378222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:43.378525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:43.379906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:43.387703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 588/1157] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:43.394099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:43.489467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a699dc9b-a267-44db-a250-43f4309fc014 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:43.527035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:43.527659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:43.527941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:43.528216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] PATH_INFO: `/v3/users/c2580191d3a947798bc83d976d1cc85f` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:43.529335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] RBAC: Authorizing `identity:get_user(user_id=c2580191d3a947798bc83d976d1cc85f)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:43.530776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:43.532379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 570/1158] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:43 2022] GET /identity/v3/users/c2580191d3a947798bc83d976d1cc85f => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:43.538764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:43.635416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f91ec8a-ab64-4432-8a0c-ad7e074dc319 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:43.668081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:43.668674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:43.668958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:43.669240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:43.679370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:43.736050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:43.737465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:43.830520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 589/1159] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:43.836498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:43.855212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:43.932981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d8f88f-cfe8-437f-9f04-d836a0428443 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:43.965129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd24af48-f675-4110-87e7-0d754ec10dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:43.965738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd24af48-f675-4110-87e7-0d754ec10dfc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:43.966018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd24af48-f675-4110-87e7-0d754ec10dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:43.966290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd24af48-f675-4110-87e7-0d754ec10dfc None admin] PATH_INFO: `/v3/projects/fd773b1b15ee4cad97121bec172aabf9` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:43.967416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd24af48-f675-4110-87e7-0d754ec10dfc None admin] RBAC: Authorizing `identity:get_project(project_id=fd773b1b15ee4cad97121bec172aabf9)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:43.968845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd24af48-f675-4110-87e7-0d754ec10dfc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:43.970426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 571/1160] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:43 2022] GET /identity/v3/projects/fd773b1b15ee4cad97121bec172aabf9 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:43.977117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:44.018018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:44.074808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ab2871-62c0-4ab3-9e8d-ff75a073d1e3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:44.108411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:44.109016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:44.109279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:44.109548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:44.110261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=fd773b1b15ee4cad97121bec172aabf9, user.id=c2580191d3a947798bc83d976d1cc85f) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:44.110568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:44.112006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b0ca6b-5273-40f8-a441-c31a57095185 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:44.117662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 590/1161] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:43 2022] GET /identity/v3/role_assignments?user.id=c2580191d3a947798bc83d976d1cc85f&scope.project.id=fd773b1b15ee4cad97121bec172aabf9&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:46.288908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405886a5-21d3-4509-99dc-1cec28ef41e6 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:46.289335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405886a5-21d3-4509-99dc-1cec28ef41e6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:46.289457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405886a5-21d3-4509-99dc-1cec28ef41e6 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:46.290775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 572/1162] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:46.295877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:46.296273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:46.296415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:46.314811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:46.316007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:46.379516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 591/1163] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:46.654629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405886a5-21d3-4509-99dc-1cec28ef41e6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:46.737984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:46.754257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405886a5-21d3-4509-99dc-1cec28ef41e6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:46.786410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:46.787038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:46.787319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:46.788123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:46.853078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:46.854516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:46.936474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:46.951899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 573/1164] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:46.957806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:46.990212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:47.054387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e932b51-6e7c-4a94-9128-8f1ef7678c31 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:47.087538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:47.088147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:47.088423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:47.088694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:47.095127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:47.096535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3990c589-039e-4f74-8bab-8195c93efb36 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:47.108048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:47.109236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:27:47.110268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 592/1165] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:27:46 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:47.115322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:47.217173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3fe5f-c5ac-46f3-9088-a18ed751424b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:47.254506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:47.255143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:47.255483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:47.255795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:47.256794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:47.257336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:47.259556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9f9fa4-76de-4ade-9416-36240ed0bfe9 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:47.267174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 574/1166] 10.222.0.20 () {56 vars in 1115 bytes} [Fri Sep 9 04:27:47 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:49.371934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517631ef-5d75-4480-b9cb-d67592fea788 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:49.372469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517631ef-5d75-4480-b9cb-d67592fea788 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:49.372469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517631ef-5d75-4480-b9cb-d67592fea788 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:49.373797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 593/1167] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:49.378912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917020bd-3b6a-42ea-baa4-b8abc7238eab None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:49.379321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917020bd-3b6a-42ea-baa4-b8abc7238eab None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:49.379491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917020bd-3b6a-42ea-baa4-b8abc7238eab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:49.397611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-917020bd-3b6a-42ea-baa4-b8abc7238eab None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:49.398785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-917020bd-3b6a-42ea-baa4-b8abc7238eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:49.443089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:49.463965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 575/1168] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:49.732257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517631ef-5d75-4480-b9cb-d67592fea788 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:49.824960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:49.833183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517631ef-5d75-4480-b9cb-d67592fea788 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:49.866189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271b08cf-204f-49a4-96ea-f9335048b93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:49.866784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b08cf-204f-49a4-96ea-f9335048b93d None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:49.867092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b08cf-204f-49a4-96ea-f9335048b93d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:49.867365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b08cf-204f-49a4-96ea-f9335048b93d None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:49.868026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271b08cf-204f-49a4-96ea-f9335048b93d None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:49.869370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271b08cf-204f-49a4-96ea-f9335048b93d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:49.884012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 594/1169] 10.222.0.20 () {60 vars in 1138 bytes} [Fri Sep 9 04:27:49 2022] POST /identity/v3/domains => generated 267 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:52.031757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f8710-7aa5-42e2-860c-9903517560ca None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:52.032299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f8710-7aa5-42e2-860c-9903517560ca None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:52.032299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f8710-7aa5-42e2-860c-9903517560ca None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:52.033646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 576/1170] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:52.038871 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:52.039311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:52.039465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:52.057667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:52.058853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:52.123751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 595/1171] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:52.388785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f8710-7aa5-42e2-860c-9903517560ca None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:52.489711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f8710-7aa5-42e2-860c-9903517560ca None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:52.521336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517841ce-c16c-4ae2-a19e-7814e49d16d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:52.522449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517841ce-c16c-4ae2-a19e-7814e49d16d6 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:52.522723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517841ce-c16c-4ae2-a19e-7814e49d16d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:52.523022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517841ce-c16c-4ae2-a19e-7814e49d16d6 None admin] PATH_INFO: `/v3/domains/c2004d438c2c48cc9357aeec246146be` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:52.532832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517841ce-c16c-4ae2-a19e-7814e49d16d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=c2004d438c2c48cc9357aeec246146be)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:52.534267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517841ce-c16c-4ae2-a19e-7814e49d16d6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:52.535857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 577/1172] 10.222.0.20 () {56 vars in 1183 bytes} [Fri Sep 9 04:27:52 2022] GET /identity/v3/domains/c2004d438c2c48cc9357aeec246146be => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:52.541760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:52.622086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:52.638615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02a37e7-4833-4b92-9cb1-41bc4f138f26 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:52.670943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c754e15d-f3d2-438c-91fd-637211dfbc54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:52.671541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754e15d-f3d2-438c-91fd-637211dfbc54 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:52.671818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754e15d-f3d2-438c-91fd-637211dfbc54 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:52.672088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754e15d-f3d2-438c-91fd-637211dfbc54 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:52.672740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c754e15d-f3d2-438c-91fd-637211dfbc54 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:52.674155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c754e15d-f3d2-438c-91fd-637211dfbc54 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:52.700145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 596/1173] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:27:52 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:54.836785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ceac2d-81fd-4304-97ba-fe41f0429f23 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:54.837334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ceac2d-81fd-4304-97ba-fe41f0429f23 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:54.837334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ceac2d-81fd-4304-97ba-fe41f0429f23 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:54.838638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 578/1174] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:54.843839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:54.844225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:54.844371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:54.862154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:54.863399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:54.927956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 597/1175] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:55.195107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ceac2d-81fd-4304-97ba-fe41f0429f23 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:55.236801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:55.294895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ceac2d-81fd-4304-97ba-fe41f0429f23 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:55.328213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befb1b0f-6749-4b56-8abc-e9dac79292ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:55.328822 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befb1b0f-6749-4b56-8abc-e9dac79292ad None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:55.329097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befb1b0f-6749-4b56-8abc-e9dac79292ad None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:55.329366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befb1b0f-6749-4b56-8abc-e9dac79292ad None admin] PATH_INFO: `/v3/domains/c2004d438c2c48cc9357aeec246146be` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:55.330461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befb1b0f-6749-4b56-8abc-e9dac79292ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=c2004d438c2c48cc9357aeec246146be)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:55.331904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befb1b0f-6749-4b56-8abc-e9dac79292ad None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:55.333436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 579/1176] 10.222.0.20 () {56 vars in 1183 bytes} [Fri Sep 9 04:27:55 2022] GET /identity/v3/domains/c2004d438c2c48cc9357aeec246146be => generated 267 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:55.339285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:55.402216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:55.437247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8393611-1448-41bf-a5b2-12b7d75729da None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:55.471929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d717c51f-0d7b-46c8-b145-790411bc22d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:55.472438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717c51f-0d7b-46c8-b145-790411bc22d6 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:55.472714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717c51f-0d7b-46c8-b145-790411bc22d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:55.472982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717c51f-0d7b-46c8-b145-790411bc22d6 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:55.473630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717c51f-0d7b-46c8-b145-790411bc22d6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:55.475108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717c51f-0d7b-46c8-b145-790411bc22d6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:55.494865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 598/1177] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:27:55 2022] POST /identity/v3/users => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:27:57.634463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ae5ff-27de-43e2-a013-7c8a6d5c8cdb None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:57.635127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ae5ff-27de-43e2-a013-7c8a6d5c8cdb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:57.635127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ae5ff-27de-43e2-a013-7c8a6d5c8cdb None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:57.636272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 580/1178] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:27:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:27:57.641437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:57.641827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:57.641990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:57.660366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:27:57.661550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:27:57.726737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 599/1179] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:27:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:27:57.997860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ae5ff-27de-43e2-a013-7c8a6d5c8cdb None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:58.094276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ae5ff-27de-43e2-a013-7c8a6d5c8cdb None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:58.103559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:58.127884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:58.128493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:58.128768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:58.129032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:58.138764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:27:58.143478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:27:58.144550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:27:58.145562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 581/1180] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:27:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:27:58.150469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:58.203221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:58.250172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2ef198-b69f-4200-9ce7-580ff654a5b8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:58.289585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:58.290190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:58.290466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:58.290743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:58.291439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:58.291750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:58.293271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:58.303681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 600/1181] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:27:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:58.309849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:58.408952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d70a4-4fa7-4c39-b4f3-384880664150 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:58.443301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:58.443925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:58.444205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:58.444479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] PATH_INFO: `/v3/users/c6e84a6a5942442a8d5238f2c1fb751f` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:58.457191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] RBAC: Authorizing `identity:get_user(user_id=c6e84a6a5942442a8d5238f2c1fb751f)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:58.458640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:58.460806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 582/1182] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:27:58 2022] GET /identity/v3/users/c6e84a6a5942442a8d5238f2c1fb751f => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:58.466706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:58.563531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3008b427-c4ff-48c1-aa72-629199e2d38d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:58.576952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:58.597027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:58.597629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:58.597909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:58.598183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:58.663583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:58.665017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:58.759253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 601/1183] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:27:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:27:58.765171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:58.792024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:58.863192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e57177e-84b5-4201-957b-962cff4e15cf None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:58.897409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1972289-d8ce-4ba8-99e7-0d70182e06f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:58.898024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1972289-d8ce-4ba8-99e7-0d70182e06f2 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:58.898298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1972289-d8ce-4ba8-99e7-0d70182e06f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:58.898565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1972289-d8ce-4ba8-99e7-0d70182e06f2 None admin] PATH_INFO: `/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:58.899689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1972289-d8ce-4ba8-99e7-0d70182e06f2 None admin] RBAC: Authorizing `identity:get_project(project_id=4ac2efe0c24c4bf4aa7bb155dd3a84b1)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:58.901105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1972289-d8ce-4ba8-99e7-0d70182e06f2 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:58.902698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 583/1184] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:27:58 2022] GET /identity/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:27:58.909200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:27:58.924066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:27:59.006999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b0c0e-c759-448c-bb8d-e1d85f2a1488 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:27:59.040423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:27:59.041012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:27:59.041289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:27:59.041558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:27:59.042266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=4ac2efe0c24c4bf4aa7bb155dd3a84b1, user.id=c6e84a6a5942442a8d5238f2c1fb751f) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:27:59.042568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:27:59.044001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a95d9-b234-4d85-9fa3-6cf88bd4bd41 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:27:59.049110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 602/1185] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:27:58 2022] GET /identity/v3/role_assignments?user.id=c6e84a6a5942442a8d5238f2c1fb751f&scope.project.id=4ac2efe0c24c4bf4aa7bb155dd3a84b1&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:01.131915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9b39da-d924-44f3-91b8-05b3be137452 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:01.132463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9b39da-d924-44f3-91b8-05b3be137452 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:01.132463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9b39da-d924-44f3-91b8-05b3be137452 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:01.133771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 584/1186] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:28:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:28:01.138981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:01.139423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:01.139528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:01.157707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:28:01.158869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:28:01.223909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 603/1187] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:28:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:28:01.487710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9b39da-d924-44f3-91b8-05b3be137452 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:01.580029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:01.588146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9b39da-d924-44f3-91b8-05b3be137452 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:01.620818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:01.621430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:01.621706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:01.621975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:01.632318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:28:01.637011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:28:01.638177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:28:01.639493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 585/1188] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:28:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:28:01.644160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:01.654066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:01.740130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5d8a87-4a54-42d0-a82d-e6fc01602b8b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:01.772634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:01.773224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:01.773501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:01.773770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:01.774421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:28:01.774725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:01.776109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:01.784874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 604/1189] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:28:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:01.791170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:01.886590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437fa9fa-b057-4ea3-b215-c6c7706a561b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:01.918988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:01.919590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:01.919863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:01.920128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] PATH_INFO: `/v3/users/c6e84a6a5942442a8d5238f2c1fb751f` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:01.921214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] RBAC: Authorizing `identity:get_user(user_id=c6e84a6a5942442a8d5238f2c1fb751f)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:01.922629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:01.924186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 586/1190] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:28:01 2022] GET /identity/v3/users/c6e84a6a5942442a8d5238f2c1fb751f => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:01.930375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:01.997234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:02.027425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa43c75a-eca9-47c0-b63b-58a804061e86 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:02.059496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da45273-7258-4219-905b-ebe921022e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:02.060104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da45273-7258-4219-905b-ebe921022e95 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:02.060379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da45273-7258-4219-905b-ebe921022e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:02.060647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da45273-7258-4219-905b-ebe921022e95 None admin] PATH_INFO: `/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:02.061754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da45273-7258-4219-905b-ebe921022e95 None admin] RBAC: Authorizing `identity:get_project(project_id=4ac2efe0c24c4bf4aa7bb155dd3a84b1)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:02.063170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da45273-7258-4219-905b-ebe921022e95 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:02.064744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 605/1191] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:28:01 2022] GET /identity/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:02.070846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:02.094659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:02.166922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af71a5e-0124-4c53-a8b9-0b6f8b7c59db None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:02.198630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b3e53-725d-47f2-87d8-55f604d7a1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:02.199292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b3e53-725d-47f2-87d8-55f604d7a1d6 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:02.199575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b3e53-725d-47f2-87d8-55f604d7a1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:02.199846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b3e53-725d-47f2-87d8-55f604d7a1d6 None admin] PATH_INFO: `/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1/users/c6e84a6a5942442a8d5238f2c1fb751f/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:02.201683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b3e53-725d-47f2-87d8-55f604d7a1d6 None admin] RBAC: Authorizing `identity:create_grant(project_id=4ac2efe0c24c4bf4aa7bb155dd3a84b1, user_id=c6e84a6a5942442a8d5238f2c1fb751f, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:02.203115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b3e53-725d-47f2-87d8-55f604d7a1d6 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:02.215508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 587/1192] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:28:02 2022] PUT /identity/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1/users/c6e84a6a5942442a8d5238f2c1fb751f/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:28:04.363129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb8fe7-2428-44e3-b73a-2504c4537b29 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:04.363665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb8fe7-2428-44e3-b73a-2504c4537b29 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:04.363665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb8fe7-2428-44e3-b73a-2504c4537b29 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:04.365486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 606/1193] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:28:04.370805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:04.371220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:04.371403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:04.389695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:28:04.390872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:28:04.455015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 588/1194] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:28:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:28:04.720383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bb8fe7-2428-44e3-b73a-2504c4537b29 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:04.819873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bb8fe7-2428-44e3-b73a-2504c4537b29 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:04.852399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:04.853016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:04.853296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:04.853568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:04.860251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:04.864748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:28:04.870310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:28:04.871479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:28:04.872499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 607/1195] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:28:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:28:04.877429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:04.891116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:04.973184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8f4944-57fe-4a4a-b113-61146473e973 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:05.006258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:05.006846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:05.007148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:05.007418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:05.008072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:28:05.008371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:05.009730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:05.017393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 589/1196] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:28:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:05.023817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:05.121575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54741a25-f68c-4efb-96ba-6066eb411b13 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:05.153670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:05.154276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:05.154557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:05.154827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] PATH_INFO: `/v3/users/c6e84a6a5942442a8d5238f2c1fb751f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:05.155969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] RBAC: Authorizing `identity:get_user(user_id=c6e84a6a5942442a8d5238f2c1fb751f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:05.157380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:05.158914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 608/1197] 10.222.0.20 () {56 vars in 1177 bytes} [Fri Sep 9 04:28:05 2022] GET /identity/v3/users/c6e84a6a5942442a8d5238f2c1fb751f => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:05.165021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:05.236888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:05.260733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b39d2-a3bb-4fb8-9bca-9d3963a2c40a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:05.293215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:05.293811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:05.294086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:05.294353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:05.357522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:05.358972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:05.427594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:05.452606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 590/1198] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:28:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:28:05.458480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:05.524889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:05.558578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5e4365-1cc3-4460-bcd9-a6ecfcc6466a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:05.591589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446669c-e641-49ec-993c-8c1fc6ba7d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:05.592203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446669c-e641-49ec-993c-8c1fc6ba7d43 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:05.592479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446669c-e641-49ec-993c-8c1fc6ba7d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:05.592749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446669c-e641-49ec-993c-8c1fc6ba7d43 None admin] PATH_INFO: `/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:05.593843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e446669c-e641-49ec-993c-8c1fc6ba7d43 None admin] RBAC: Authorizing `identity:get_project(project_id=4ac2efe0c24c4bf4aa7bb155dd3a84b1)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:05.595289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e446669c-e641-49ec-993c-8c1fc6ba7d43 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:05.596885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 609/1199] 10.222.0.20 () {56 vars in 1186 bytes} [Fri Sep 9 04:28:05 2022] GET /identity/v3/projects/4ac2efe0c24c4bf4aa7bb155dd3a84b1 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:05.603109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:28:05.697311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0320e53-1259-461e-b1e3-901b312e650d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:28:05.729552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:28:05.730135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:28:05.730412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:28:05.730688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:28:05.731426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=4ac2efe0c24c4bf4aa7bb155dd3a84b1, user.id=c6e84a6a5942442a8d5238f2c1fb751f) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:28:05.731730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:28:05.733139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0c91f8-0d96-48bb-8821-c087a61e0308 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:28:05.738611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 591/1200] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:28:05 2022] GET /identity/v3/role_assignments?user.id=c6e84a6a5942442a8d5238f2c1fb751f&scope.project.id=4ac2efe0c24c4bf4aa7bb155dd3a84b1&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:28:15.587625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=33 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:28:15.736162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=81 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:01.199593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e1a90-6104-4c1b-b2da-8a78a5de6490 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:01.200236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e1a90-6104-4c1b-b2da-8a78a5de6490 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:01.200236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e1a90-6104-4c1b-b2da-8a78a5de6490 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:01.201493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 610/1201] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:01.207148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:01.207716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:01.207716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:01.229418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:01.230639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:01.327084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 592/1202] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:01.607733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4e1a90-6104-4c1b-b2da-8a78a5de6490 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:01.743700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4e1a90-6104-4c1b-b2da-8a78a5de6490 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:01.797726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:01.798463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:01.798756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:01.799126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:01.812914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:01.814584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:01.831541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:01.832771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:01.836227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:01.836227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:01.836227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:01.836227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 611/1203] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:01.839341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:01.967037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce7810-8036-4a62-9078-3ab9a38f23f1 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:02.005645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:02.012698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:02.013368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:02.013652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:02.013928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:02.014648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:02.014988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:02.016531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988569bc-480b-4d1f-a116-622199e2a654 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:02.029567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 593/1204] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:29:02.035701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:02.109497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:02.193914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09617b4b-61e4-4bb1-8450-fb807af10ec0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:02.239956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce99eb-8a60-4359-a3c8-a9f79c4fde0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:02.241503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce99eb-8a60-4359-a3c8-a9f79c4fde0d None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:02.242054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce99eb-8a60-4359-a3c8-a9f79c4fde0d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:02.242568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce99eb-8a60-4359-a3c8-a9f79c4fde0d None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:02.243440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce99eb-8a60-4359-a3c8-a9f79c4fde0d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:02.245449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce99eb-8a60-4359-a3c8-a9f79c4fde0d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:02.283935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 612/1205] 10.222.0.20 () {60 vars in 1132 bytes} [Fri Sep 9 04:29:02 2022] POST /identity/v3/users => generated 251 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:29:04.702185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b14b34-6666-4a7f-9e25-f38ecdad277f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:04.702185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b14b34-6666-4a7f-9e25-f38ecdad277f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:04.702185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b14b34-6666-4a7f-9e25-f38ecdad277f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:04.702185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 594/1206] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:04.708500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:04.709119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:04.709119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:04.727706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:04.728883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:04.805319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 613/1207] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:05.078859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b14b34-6666-4a7f-9e25-f38ecdad277f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:05.215859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b14b34-6666-4a7f-9e25-f38ecdad277f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:05.263673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:05.264346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:05.264693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:05.264996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:05.279466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:29:05.285329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:05.286353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:05.287322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 595/1208] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:29:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:05.293723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:05.342673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:05.417911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceab3c2-99d1-46b0-81ef-8d0e0afcd91d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:05.461514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:05.462085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:05.462337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:05.462675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:05.463375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:05.463680 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:05.465224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:05.476505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 614/1209] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:29:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:05.481465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:05.544065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:05.619199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5d88c6-ee7a-4104-a594-4168325d8d23 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:05.672827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:05.673635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:05.673950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:05.674257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:05.687112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:05.688732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:05.705217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:05.706758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:05.708050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:05.708050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:05.708050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:05.711097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 596/1210] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:05.712751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:05.823929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:05.827776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3bffdb-4d33-4b8f-ab14-c77c53cc7dd6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:05.866910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:05.867606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:05.867892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:05.868205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:05.868907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:05.869283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:05.870666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:05.884217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 615/1211] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:29:05.891101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:06.002267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b18c35-876d-4b0a-82d5-f3207962d7f3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:06.004523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:06.042798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:06.043455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:06.043733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:06.044161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:06.052666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:06.054104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:29:06.066418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:06.067511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:06.068732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 597/1212] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:05 2022] GET /identity/v3/users/placement => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:06.076588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:06.193273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15299bc9-c8dc-446f-bb39-a88191952596 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:06.232987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:06.233777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:06.234177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:06.234574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:06.235521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:06.235983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:06.238159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:06.263388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 616/1213] 10.222.0.20 () {56 vars in 1143 bytes} [Fri Sep 9 04:29:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:06.270058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:06.392806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946900f1-0c83-42ae-a7cc-47842f8e8a8e None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:06.434580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:06.435254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:06.435537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:06.435808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:06.475293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:06.524121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:06.525826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:06.662078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 598/1214] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:29:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:29:06.668700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:06.693549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:06.798693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f8530-60fb-4c66-a5ad-9b88c9de4586 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:06.843193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:06.843877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:06.844312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:06.844587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:06.856159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:06.857641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:06.873077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:06.878954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:06.880093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:06.880093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:06.880093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:06.880093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 617/1215] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:06.880371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:06.950659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:06.995653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d12464-500e-4aae-a3cd-26d0d67cad9d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:07.037941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:07.038746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:07.039082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:07.039355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:07.040230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:07.040684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:07.045441 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:07.061574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 599/1216] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:07.062696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:07.148000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:07.188310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb2a27d-a269-45cb-bcd1-fbe6cdd1a5cd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:07.223895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:07.224623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:07.224909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:07.225244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:07.232635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:07.234248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:29:07.247322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:07.248498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:07.249616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 618/1217] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:29:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:07.255074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:07.368728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbb2148-de58-4d17-b081-72622b9c20f7 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:07.393381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:07.405754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:07.406358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:07.406643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:07.406914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:07.407627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:07.407933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:07.410193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5e3ab-52f4-40b0-a16d-09cff98a3c57 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:07.426296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 600/1218] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:29:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:29:07.434522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:07.553190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3bcfdb-850e-46eb-99cc-2beda0cc0f57 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:07.591966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:07.599661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:07.600349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:07.600598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:07.600880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:07.601637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=7ef39450c46b43a5bed04645689610f6) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:07.601974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:07.603592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fea19e-030e-4b11-8ac5-922a9fae90a2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:07.610112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 619/1219] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:29:07 2022] GET /identity/v3/role_assignments?user.id=7ef39450c46b43a5bed04645689610f6&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:09.779206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7f91b-3864-41e5-a417-742e18d0e8a8 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:09.779829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7f91b-3864-41e5-a417-742e18d0e8a8 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:09.779829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7f91b-3864-41e5-a417-742e18d0e8a8 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:09.781349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 601/1220] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:09.787609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:09.788049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:09.788160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:09.809375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:09.810660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:09.885352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 620/1221] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:10.198836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7f91b-3864-41e5-a417-742e18d0e8a8 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:10.319029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7f91b-3864-41e5-a417-742e18d0e8a8 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:10.356180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:10.356811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:10.357093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:10.357368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:10.371523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:29:10.377399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:10.378387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:29:10.379272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:10.380108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 602/1222] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:29:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:10.385884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:10.497193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d52525-3e1e-4fee-a6fb-d9c5f04fa232 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:10.523249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:10.536080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:10.536688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:10.536968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:10.537243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:10.537905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:10.538208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:10.539856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:10.548966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 621/1223] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:29:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:10.555246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:10.614924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:10.677691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ddd37d-42c4-44ce-a89b-47b5cb649263 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:10.714089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:10.714798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:10.715119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:10.715476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:10.730921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:10.732596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:10.751099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:10.752238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:10.753362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:10.753362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:10.753362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:10.753362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 603/1224] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:10.758457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:10.865831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9892c3fc-9c6c-4ee2-b058-791bc3bcc498 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:10.905754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:10.906492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:10.906876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:10.907283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:10.907924 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:10.908286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:10.910489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:10.924602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 622/1225] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:10.930751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:11.041884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:11.050650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40d7d1d-23fe-4e8b-a3c9-cbdccedaeec3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:11.087248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:11.087938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:11.088252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:11.088553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:11.096547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:11.098571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:29:11.110976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:11.112344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:11.113319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 604/1226] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:11.118083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:11.178251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:11.233511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9e7628-c0e7-4e86-a086-c0349bdf454d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:11.273670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:11.274315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:11.274595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:11.274904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:11.275707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:11.276081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:11.277556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:11.300633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 623/1227] 10.222.0.20 () {56 vars in 1143 bytes} [Fri Sep 9 04:29:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:11.301773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:11.408119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09dcfe-dbcf-49c1-919e-26aa4902eeee None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:11.447306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:11.447941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:11.448236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:11.448590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:11.459728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:11.461232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:11.478025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:11.479578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:11.480691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:11.480691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:11.480691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:11.480691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 605/1228] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:11.485421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:11.608980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d79ec0-8443-4faa-b0b9-3c01a7fe7eb4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:11.614437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:11.655155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:11.655771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:11.656053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:11.656327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:11.656975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:11.657281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:11.658643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:11.670207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 624/1229] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:29:11.676464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:11.686811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:11.787950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc39d861-6492-4a71-b37a-bd99774e1e96 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:11.823058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:11.823833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:11.824117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:11.824444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:11.831398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:11.832851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:29:11.844083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:11.845038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:11.845947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 606/1230] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:29:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:11.851589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:11.964256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9772ac2-93a7-47cf-95e3-020b0f5864bc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:12.002668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:12.003919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:12.004198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:12.004467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:12.005159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:12.005463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:12.006884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e09762-868e-4329-b7ea-39109ed51e53 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:12.019154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 625/1231] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:29:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:12.025867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:12.100349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:12.139171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70efc504-8e40-4187-91ea-41482a144f4a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:12.174374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377d5a6-a073-4fd6-8e70-e8f531167776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:12.175682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377d5a6-a073-4fd6-8e70-e8f531167776 None admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:12.175960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377d5a6-a073-4fd6-8e70-e8f531167776 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:12.176233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377d5a6-a073-4fd6-8e70-e8f531167776 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/7ef39450c46b43a5bed04645689610f6/roles/f2705e4bdf8148c9961220bef051b0f3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:12.190643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b377d5a6-a073-4fd6-8e70-e8f531167776 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=7ef39450c46b43a5bed04645689610f6, role_id=f2705e4bdf8148c9961220bef051b0f3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:12.192121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b377d5a6-a073-4fd6-8e70-e8f531167776 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:12.205985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 607/1232] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:29:12 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/7ef39450c46b43a5bed04645689610f6/roles/f2705e4bdf8148c9961220bef051b0f3 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:29:14.670584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca94f06f-cd2a-4345-85b0-148395b3130a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:14.671195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca94f06f-cd2a-4345-85b0-148395b3130a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:14.671318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca94f06f-cd2a-4345-85b0-148395b3130a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:14.672824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 626/1233] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:14.678268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:14.678740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:14.678849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:14.698188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:14.699445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:14.772904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 608/1234] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:15.043209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca94f06f-cd2a-4345-85b0-148395b3130a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:15.074640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:15.162137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca94f06f-cd2a-4345-85b0-148395b3130a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:15.200907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:15.201605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:15.201923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:15.202224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:15.213886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:29:15.219524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:15.220659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:15.221511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 627/1235] 10.222.0.20 () {56 vars in 1102 bytes} [Fri Sep 9 04:29:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:29:15.226878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:15.281879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:15.341784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db1e532-ba90-44b4-947d-18e929136623 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:15.376704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:15.377329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:15.377580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:15.378354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:15.379040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] RBAC: Adding query filter params (name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:15.379343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:15.380710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:15.389799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 609/1236] 10.222.0.20 () {56 vars in 1103 bytes} [Fri Sep 9 04:29:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:15.396573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:15.488382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:15.506429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1716f207-53b3-4369-8c9d-0a831409a41f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:15.544467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:15.545091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:15.545369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:15.545644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:15.558206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:15.559810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:15.576772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:15.578379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:15.579657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:15.579657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:15.579657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:15.580027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 628/1237] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:15.586973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:15.698769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee431d75-8f1f-4dfb-b121-2b013bae8a80 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:15.709668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:15.738071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:15.738734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:15.739085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:15.739361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:15.740081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:15.740335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:15.741776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:15.755260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 610/1238] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:15.761147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:15.872061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fcb7d3-a825-418c-aa42-91bcf2d16b4b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:15.913062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:15.914098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:15.914633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:15.915060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:15.918892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:15.927483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:15.930208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:29:15.945805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:15.947090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:15.948452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:15.950465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 629/1239] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:15.956270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:16.074316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc67abbc-1bfc-4c8d-8e59-ecd9a35f355a None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:16.111803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:16.113023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:16.113305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:16.113580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:16.114362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:16.114719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:16.116442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:16.135550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 611/1240] 10.222.0.20 () {56 vars in 1143 bytes} [Fri Sep 9 04:29:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:16.141504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:16.257090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d46f8a-5f1d-44d3-bebf-79a5c303d4c1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:16.294065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:16.294693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:16.294981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:16.295257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:16.359814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:16.368018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:16.369844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:16.482243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 630/1241] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:29:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:29:16.488317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:16.522410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:16.621452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b954023b-eaf8-46d1-996a-0d7b32462079 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:16.667718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:16.668353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:16.668631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:16.668904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:16.679517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:16.681006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:16.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:16.701119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:16.702061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:16.702061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:16.702061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:16.706973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 612/1242] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:16.707923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:16.820378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:16.826712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497cd4b6-c05b-49e3-8a7d-b31118f97355 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:16.866081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:16.866719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:16.867043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:16.867283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:16.867921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:16.868220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:16.869597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:16.881875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 631/1243] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:16.887723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:16.973113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:16.996306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928bf49-9a56-4c7c-b48e-53721dfe69e3 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:17.036223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:17.036856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:17.037149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:17.037410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:17.044398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:17.045836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:29:17.058596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:17.059777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:17.060888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 613/1244] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:29:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 09 04:29:17.068300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:17.175928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c21a21-8c8f-41dd-8238-6091e789067a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:17.213486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:17.214398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:17.214686 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:17.214989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:17.215710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:17.216110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:17.217999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469dbdd-6821-43e8-9467-25efeb047056 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:17.235556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 632/1245] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:29:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:17.242046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:17.364187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e605e1-81c9-4e74-99b0-3cfb6cacde3c None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:17.408870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:17.409997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:17.410272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:17.410537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:17.411314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] RBAC: Adding query filter params (role.id=f2705e4bdf8148c9961220bef051b0f3, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=7ef39450c46b43a5bed04645689610f6) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:17.411621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:17.413063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45431de7-f8d7-4acb-b799-1f44ad0fe85a None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:17.419609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 614/1246] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:29:17 2022] GET /identity/v3/role_assignments?user.id=7ef39450c46b43a5bed04645689610f6&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=f2705e4bdf8148c9961220bef051b0f3 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:19.788822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5732ac6-05a7-4ff5-94ea-4dd5154557bf None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:19.789361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5732ac6-05a7-4ff5-94ea-4dd5154557bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:19.789361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5732ac6-05a7-4ff5-94ea-4dd5154557bf None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:19.790873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 633/1247] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:29:19.796849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:19.797432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:19.797561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:19.804989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:19.821163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:19.822384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:19.900197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 615/1248] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:20.210282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5732ac6-05a7-4ff5-94ea-4dd5154557bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:20.230617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:20.323997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5732ac6-05a7-4ff5-94ea-4dd5154557bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:20.362689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:20.363433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:20.363738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:20.364050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:20.376349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:29:20.381872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:20.382915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:20.383983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 634/1249] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:29:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:20.389619 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:20.494195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f95036-b29c-4e8c-b952-83bdc82f213a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:20.523808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:20.529738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:20.530340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:20.530618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:20.530883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:20.531605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:20.531906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:20.533304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:20.541478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 616/1250] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:29:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:20.547571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:20.637399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:20.661699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3981b9b-b524-4e20-8203-8ab47acfd118 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:20.696028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:20.696900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:20.697181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:20.697454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:20.708795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:20.710976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:20.724353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:20.725508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:20.726753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:20.726753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:20.726753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:20.726753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 635/1251] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:20.731229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:20.830681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bc25b-909e-4f2e-b0c2-66a5855fc746 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:20.862504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:20.863136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:20.863416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:20.863686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:20.864316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:20.864617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:20.866014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:20.878372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 617/1252] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:20.884094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:20.980789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdaf69-911c-47f7-88c8-729d93452a3d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:21.006168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:21.013957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:21.014561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:21.014836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:21.015132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:21.022349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:21.023791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:29:21.034503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:21.035478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:21.036524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 636/1253] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:20 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:21.041309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:21.095676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:21.138780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a0796a-e433-4f9b-84ba-3849f882820d None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:21.171127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:21.171713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:21.171987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:21.172255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:21.172971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:21.173268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:21.175288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:21.191702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 618/1254] 10.222.0.20 () {56 vars in 1143 bytes} [Fri Sep 9 04:29:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:21.197363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:21.294279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d623e1-4f96-4f5a-ad3a-2f44ed945eee None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:21.327014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:21.327607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:21.327886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:21.328156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:21.381066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:21.393347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:21.394781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:21.489445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 637/1255] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:29:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:29:21.495383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:21.596079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d03177-5f1c-4d9b-b7b9-6126a5e0fefa None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:21.611592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:21.634067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:21.634660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:21.634950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:21.635241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:21.645402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:21.646834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:21.661618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:21.662745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:21.663843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:21.663843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:21.663843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:21.663843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 619/1256] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:21.668577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:21.752634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:21.764771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477e9e1e-aac9-4259-b730-4048a673bbc3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:21.798819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:21.799463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:21.799739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:21.800007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:21.800640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:21.800938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:21.802581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:21.814109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 638/1257] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:21.819718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:21.916248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf535a2-6cf6-43d6-9b39-a5e80ff4d8c1 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:21.948369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:21.948975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:21.949251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:21.949521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:21.956439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:21.957864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:29:21.968671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:21.969713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:21.970988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 620/1258] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:29:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:21.975409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:22.071128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f87abf-701d-4c13-9b11-6adc3772e15b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:22.103286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b554b771-809a-4820-abd7-77878a28708b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:22.103879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b554b771-809a-4820-abd7-77878a28708b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:22.104160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b554b771-809a-4820-abd7-77878a28708b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:22.104432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b554b771-809a-4820-abd7-77878a28708b None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:22.105116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b554b771-809a-4820-abd7-77878a28708b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:22.105418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b554b771-809a-4820-abd7-77878a28708b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:22.106838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b554b771-809a-4820-abd7-77878a28708b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:22.118437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 639/1259] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:29:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:22.124226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:22.149708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:22.220819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7424b4-4df0-4408-b4db-f772253bd5fd None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:22.252169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:22.253346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:22.253790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:22.254321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:22.254939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=7ef39450c46b43a5bed04645689610f6) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:22.255258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:22.256683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f69bde4-0922-402a-bf0a-e1d70854b0f0 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:22.262014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 621/1260] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:29:22 2022] GET /identity/v3/role_assignments?user.id=7ef39450c46b43a5bed04645689610f6&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:24.360605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702e852-dac5-4f85-83cc-a4dcffd6ed8c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:24.361156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702e852-dac5-4f85-83cc-a4dcffd6ed8c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:24.361156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702e852-dac5-4f85-83cc-a4dcffd6ed8c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:24.363050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 640/1261] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:24.368166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:24.368542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:24.368718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:24.387101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:24.388284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:24.451852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 622/1262] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:24.713483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702e852-dac5-4f85-83cc-a4dcffd6ed8c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:24.724977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:24.812614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702e852-dac5-4f85-83cc-a4dcffd6ed8c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:24.844843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:24.845448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:24.845724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:24.845994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:24.856258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:29:24.860940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:24.862069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:24.863115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 641/1263] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:29:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:24.868195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:24.885728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:24.964995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593feb2b-6e8c-4551-a180-9327eb8ad519 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:24.996529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:24.997118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:24.997393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:24.997663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:24.998314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:24.998613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.000000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.008108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 623/1264] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:29:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:25.013917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.078679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:25.110540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071c59ef-d1b6-4524-aff4-e12f72f7a95f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:25.142710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:25.143923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:25.144197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:25.144467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:25.154056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.155515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:25.169490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:25.170475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:25.171425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:25.171425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:25.171425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:25.171425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 642/1265] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:25.176258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.253935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:25.273236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e967dc0c-e604-40b4-b000-708f3c0ad282 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:25.305270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:25.305864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:25.306142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:25.306416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:25.307349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:25.307656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.309025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.320676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 624/1266] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:25.326225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.421888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ac4ee7-4650-44db-a509-ddfdec76b4fc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:25.441640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:25.455166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:25.455797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:25.456073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:25.456344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:25.463450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.464853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:29:25.475540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:25.476656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:25.477739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 643/1267] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:25.482189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.579241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a3f351-e061-4c97-890f-17b7ab7faa01 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:25.618395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:25.619038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:25.619321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:25.619590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:25.620305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:25.620604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.622011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.631145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:25.639645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 625/1268] 10.222.0.20 () {56 vars in 1143 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:25.645406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.744476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b10e2b1-6b84-4e6b-bd9c-4ff7b0b45d94 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:25.776475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:25.777090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:25.777370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:25.777646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:25.792123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.794574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:25.809267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:25.810396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:25.811729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:25.811729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:25.811729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:25.811729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 644/1269] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:25.816136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.913079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc5da3b-f80f-4e2d-ad2e-66e0a6d3e19b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:25.946050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:25.946644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:25.946924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:25.947237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:25.947876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:25.948179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:25.949545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:25.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 626/1270] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:25.966096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:25.997681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:26.064469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1abe9cb-fb14-46bb-a26c-ec9b0444e4cb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:26.096926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:26.097526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:26.097801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:26.098067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:26.104450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:26.105864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:29:26.116662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:26.117772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:26.118899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 645/1271] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:29:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:26.123605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:26.180454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:26.223223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e9640-8eab-4208-bc2c-958586195ff4 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:26.255428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:26.256032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:26.256311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:26.256582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:26.257273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:26.257581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:26.259025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e79f92-1e79-45b9-8924-bb5a930fc9ca None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:26.271127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 627/1272] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:29:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:26.277179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:26.365276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:26.374049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719868c2-db55-4846-8220-60daea50e3ea None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:26.406081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eca73c-7f04-4ca0-96a1-074ec9610db8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:26.406711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eca73c-7f04-4ca0-96a1-074ec9610db8 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:26.407018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eca73c-7f04-4ca0-96a1-074ec9610db8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:26.407295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eca73c-7f04-4ca0-96a1-074ec9610db8 None admin] PATH_INFO: `/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/7ef39450c46b43a5bed04645689610f6/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:26.409688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eca73c-7f04-4ca0-96a1-074ec9610db8 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ac32ce05067416fbc8af4a0e7da4cfe, user_id=7ef39450c46b43a5bed04645689610f6, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:26.411118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eca73c-7f04-4ca0-96a1-074ec9610db8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:26.422912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 646/1273] 10.222.0.20 () {58 vars in 1439 bytes} [Fri Sep 9 04:29:26 2022] PUT /identity/v3/projects/6ac32ce05067416fbc8af4a0e7da4cfe/users/7ef39450c46b43a5bed04645689610f6/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:29:28.518612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed20afc1-8cf2-42c5-9bd0-312352b71cdb None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:28.519297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed20afc1-8cf2-42c5-9bd0-312352b71cdb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:28.519297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed20afc1-8cf2-42c5-9bd0-312352b71cdb None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:28.520427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 628/1274] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:28.525602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:28.526016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:28.526152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:28.544634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:28.545815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:28.620711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 647/1275] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:28.887468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed20afc1-8cf2-42c5-9bd0-312352b71cdb None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:28.999223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed20afc1-8cf2-42c5-9bd0-312352b71cdb None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.015535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:29.032504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.033123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.033399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.034290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:29.043987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 09 04:29:29.048701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:29.049816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:29.050998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 629/1276] 10.222.0.20 () {56 vars in 1096 bytes} [Fri Sep 9 04:29:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:29.055802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:29.132964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:29.151381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e89d69-2007-4b85-bc05-94b0e548a025 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.183848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.184438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.184711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.184978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:29.185628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:29.185923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:29.187289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:29.194911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 648/1277] 10.222.0.20 () {56 vars in 1099 bytes} [Fri Sep 9 04:29:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:29.200593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:29.298973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bae856-d93e-4471-9eba-c0782cf672af None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.331626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.332250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.332528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.332800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:29.342358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:29.343834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:29.358436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:29.359580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:29.360635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:29.360635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:29.360635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:29.360635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 630/1278] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:29.365320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:29.462117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd88a1b-8f4f-467f-ab79-041d07a0ff15 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.493568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.494148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.494424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.494695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:29.495351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:29.495651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:29.496989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:29.508638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 649/1279] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:29.514164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:29.544016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:29.614276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b890ef5-7288-479b-9353-12761f7ce10b None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.646722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.647360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.647637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.647904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:29.654910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:29.656368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 09 04:29:29.667097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:29.668162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:29.669264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 631/1280] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:29 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:29.674154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:29.677876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:29.770563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32abedcb-4716-4fff-8ca4-387917c04188 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.803510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.804108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.804383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.804649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:29.805361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:29.805658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:29.807360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:29.824156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 650/1281] 10.222.0.20 () {56 vars in 1143 bytes} [Fri Sep 9 04:29:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:29.829787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:29.916201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:29.926316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2debb6-492e-4627-a575-a687f4c36473 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:29.960988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:29.961569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:29.961844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:29.962114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:30.026511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:30.027914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:30.118376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:30.129291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 632/1282] 10.222.0.20 () {58 vars in 1282 bytes} [Fri Sep 9 04:29:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 09 04:29:30.135074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:30.195068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:30.232429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad7bdf-52d6-4b28-a795-2c3d06df835c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:30.267346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:30.267973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:30.268252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:30.268528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:30.278388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:30.279806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 09 04:29:30.294410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:30.295555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 09 04:29:30.296479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 09 04:29:30.296479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 09 04:29:30.296479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:30.296479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 651/1283] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:29:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:30.301756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:30.395759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cf7e0e-54f7-4aeb-890f-3f981845b258 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:30.428696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:30.429288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:30.429552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:30.429824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:30.430450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:30.430749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:30.432397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:30.443070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 633/1284] 10.222.0.20 () {56 vars in 1109 bytes} [Fri Sep 9 04:29:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:30.448535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:30.546972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaecd9be-d6bc-48f9-a3c7-f86133276359 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:30.560402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:30.581880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:30.582485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:30.582762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:30.583061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:30.589591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:30.591037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 09 04:29:30.602442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:30.603540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Sep 09 04:29:30.604805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 652/1285] 10.222.0.20 () {56 vars in 1111 bytes} [Fri Sep 9 04:29:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:30.609252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:30.657462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:30.706726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc12a685-f4fc-487f-9ff2-c866c0f4ea83 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:30.739405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:30.739996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:30.740278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:30.740555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:30.741250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:30.741556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:30.742998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68582be-ffe3-45f7-9ff0-e8e8baa40954 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:30.754548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 634/1286] 10.222.0.20 () {56 vars in 1148 bytes} [Fri Sep 9 04:29:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:30.760462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:30.857306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac3a7f2-e773-4545-be3b-905bef8fc281 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:30.888921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:30.889508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:30.889787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:30.890058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:30.890773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] RBAC: Adding query filter params (role.id=a5062e61497c4d40b35fe67595be3e2e, scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe, user.id=7ef39450c46b43a5bed04645689610f6) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:30.891111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:30.892554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd56fce-a1dd-4973-a038-e0f24d3ecd93 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:30.898666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 653/1287] 10.222.0.20 () {56 vars in 1374 bytes} [Fri Sep 9 04:29:30 2022] GET /identity/v3/role_assignments?user.id=7ef39450c46b43a5bed04645689610f6&scope.project.id=6ac32ce05067416fbc8af4a0e7da4cfe&role.id=a5062e61497c4d40b35fe67595be3e2e => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:33.061070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa9ee0-58c2-4637-aa64-95b5aa8b3109 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:33.061610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa9ee0-58c2-4637-aa64-95b5aa8b3109 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:33.061610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa9ee0-58c2-4637-aa64-95b5aa8b3109 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:33.062981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 635/1288] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:33.068170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:33.068549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:33.068712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:33.087156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:33.088325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:33.116863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:33.152495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 654/1289] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:33.417990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa9ee0-58c2-4637-aa64-95b5aa8b3109 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:33.514875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa9ee0-58c2-4637-aa64-95b5aa8b3109 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:33.525606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:33.548109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:33.548723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:33.549536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:33.549813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:33.550389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:33.551792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:33.560518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:33.561809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:33.562760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 636/1290] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:29:33 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:33.568158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:33.687460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a0d9e-f253-45cd-89a9-13fbc3faed0c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:33.723030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:33.723626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:33.723901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:33.724169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:33.724785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] RBAC: Adding query filter params (name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:33.725087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:33.726436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0d7d98-c8bb-41a4-9b9c-17272fbcfeed None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:33.732043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 655/1291] 10.222.0.20 () {56 vars in 1116 bytes} [Fri Sep 9 04:29:33 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:33.738295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:33.835626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb019264-90e5-4e00-af9d-57d3b697b0c8 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:33.868089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:33.868685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:33.868965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:33.869237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] PATH_INFO: `/v3/services` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:33.869861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:33.870167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:33.871564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a0e75-f9c6-4c0f-b521-b0079052af75 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:33.875834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 637/1292] 10.222.0.20 () {56 vars in 1116 bytes} [Fri Sep 9 04:29:33 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:36.001328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde5415-3224-41dc-83b4-bb15006a7a95 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:36.001864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde5415-3224-41dc-83b4-bb15006a7a95 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:36.001864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde5415-3224-41dc-83b4-bb15006a7a95 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:36.003256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 656/1293] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:36.008321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabc16d6-db01-4644-8010-e20076fff21f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:36.008704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabc16d6-db01-4644-8010-e20076fff21f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:36.008879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabc16d6-db01-4644-8010-e20076fff21f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:36.027263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-fabc16d6-db01-4644-8010-e20076fff21f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:36.028453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-fabc16d6-db01-4644-8010-e20076fff21f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:36.057314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:36.093249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 638/1294] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:36.358870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde5415-3224-41dc-83b4-bb15006a7a95 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:36.389319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:36.457602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde5415-3224-41dc-83b4-bb15006a7a95 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:36.489650 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1630a4-829c-4a7e-8bc7-eb9acb922a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:36.490235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1630a4-829c-4a7e-8bc7-eb9acb922a93 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:36.490513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1630a4-829c-4a7e-8bc7-eb9acb922a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:36.490789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1630a4-829c-4a7e-8bc7-eb9acb922a93 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:36.491457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1630a4-829c-4a7e-8bc7-eb9acb922a93 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:36.492799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1630a4-829c-4a7e-8bc7-eb9acb922a93 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:36.507904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 657/1295] 10.222.0.20 () {60 vars in 1141 bytes} [Fri Sep 9 04:29:36 2022] POST /identity/v3/services => generated 246 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:29:38.650742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a020c43-e15b-4f64-870e-18fd6675e9b3 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:38.651304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a020c43-e15b-4f64-870e-18fd6675e9b3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:38.651304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a020c43-e15b-4f64-870e-18fd6675e9b3 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:38.652641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 639/1296] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:38.657867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:38.658848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:38.658978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:38.677335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:38.678511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:38.748853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 658/1297] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:38 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:39.024594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a020c43-e15b-4f64-870e-18fd6675e9b3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:39.121698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a020c43-e15b-4f64-870e-18fd6675e9b3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:39.154189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:39.154825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:39.155134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:39.155411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:39.156193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:39.157565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:39.169205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:39.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:39.171689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 640/1298] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:29:39 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:39.176332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:39.194856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:39.274891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb59642-ca7d-493a-b81f-f06746d089af None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:39.307128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:39.307724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:39.308002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:39.308272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:39.308889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:39.309189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:39.310538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:39.316004 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 659/1299] 10.222.0.20 () {56 vars in 1116 bytes} [Fri Sep 9 04:29:39 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:39.321851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:39.353459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:39.420253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a715210-21db-4770-a5f1-d42ea4084c38 None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:39.452777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:39.453367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:39.453646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:39.453920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:39.454610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] RBAC: Adding query filter params (interface=public, service_id=332d1f89329b4822a91d31f1f038054f, region_id=RegionOne) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:39.454918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:39.456842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece8817-cc95-44e1-b586-7fda0d8096bc None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:39.461677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 641/1300] 10.222.0.20 () {56 vars in 1251 bytes} [Fri Sep 9 04:29:39 2022] GET /identity/v3/endpoints?service_id=332d1f89329b4822a91d31f1f038054f&interface=public®ion_id=RegionOne => generated 194 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:39.467271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:39.545390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:39.563673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af81c6ee-bffd-46c4-9c84-bec56e6aae17 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:39.600851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd3dd46-4499-4b24-8a7c-e2f726fa7948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:39.601544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd3dd46-4499-4b24-8a7c-e2f726fa7948 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:39.601830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd3dd46-4499-4b24-8a7c-e2f726fa7948 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:39.602116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd3dd46-4499-4b24-8a7c-e2f726fa7948 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:39.602720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd3dd46-4499-4b24-8a7c-e2f726fa7948 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:39.604129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd3dd46-4499-4b24-8a7c-e2f726fa7948 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:39.611723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 660/1301] 10.222.0.20 () {56 vars in 1087 bytes} [Fri Sep 9 04:29:39 2022] GET /identity/v3/services => generated 2191 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:29:41.725031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0810be41-c8ed-42bc-9bfb-26cf8c8551f0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:41.725601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0810be41-c8ed-42bc-9bfb-26cf8c8551f0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:41.725601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0810be41-c8ed-42bc-9bfb-26cf8c8551f0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:41.726917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 642/1302] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:29:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:29:41.732118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:41.732569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:41.732680 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:41.751041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:29:41.752249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:29:41.818220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 661/1303] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:29:41 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 09 04:29:42.084306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0810be41-c8ed-42bc-9bfb-26cf8c8551f0 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:42.177128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:42.184445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0810be41-c8ed-42bc-9bfb-26cf8c8551f0 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:42.217167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:42.217776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:42.218054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:42.218324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] PATH_INFO: `/v3/services/placement` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:42.218898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:42.220283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 09 04:29:42.228713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 09 04:29:42.229883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.get_or_create( Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application with Lock( Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self._enter() Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application return self.creator() Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application created_value = creator( Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: ERROR keystone.server.flask.application Sep 09 04:29:42.231030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 643/1304] 10.222.0.20 () {56 vars in 1117 bytes} [Fri Sep 9 04:29:42 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 09 04:29:42.235794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:42.309973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:42.333331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9172429-25ed-43e3-af24-631bd492cc91 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:42.365775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:42.366315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:42.366594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:42.366867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:42.367519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:29:42.367822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:42.369172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0057f301-2b6a-477b-8679-8fdec714431f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:42.374151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 662/1305] 10.222.0.20 () {56 vars in 1116 bytes} [Fri Sep 9 04:29:42 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:29:42.380323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:29:42.475942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a34b7-3c74-4f03-8dac-2c703d3351ef None admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:29:42.508121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2173df67-4eb5-494e-a6a6-19d1fa5c4198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:29:42.508714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2173df67-4eb5-494e-a6a6-19d1fa5c4198 None admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:29:42.508990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2173df67-4eb5-494e-a6a6-19d1fa5c4198 None admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:29:42.509260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2173df67-4eb5-494e-a6a6-19d1fa5c4198 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:29:42.509912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2173df67-4eb5-494e-a6a6-19d1fa5c4198 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:29:42.511286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2173df67-4eb5-494e-a6a6-19d1fa5c4198 None admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:29:42.530590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 644/1306] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:29:42 2022] POST /identity/v3/endpoints => generated 331 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:29:52.371341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=30 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:29:52.526987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:46.540317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c1d5a-3d05-4519-8451-d9b89324b588 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:46.540811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c1d5a-3d05-4519-8451-d9b89324b588 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:46.540896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c1d5a-3d05-4519-8451-d9b89324b588 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:46.548005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f8935-e883-418b-aaa1-d0f2612d5c8c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:46.548435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f8935-e883-418b-aaa1-d0f2612d5c8c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:46.548575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f8935-e883-418b-aaa1-d0f2612d5c8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:46.553394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 663/1307] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:30:46 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:30:46.677573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-744f8935-e883-418b-aaa1-d0f2612d5c8c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:30:46.679280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-744f8935-e883-418b-aaa1-d0f2612d5c8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:46.799403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 645/1308] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:30:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:30:47.091586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d01ff-ae97-474e-bc11-471101d3268e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:47.091876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d01ff-ae97-474e-bc11-471101d3268e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:47.092155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d01ff-ae97-474e-bc11-471101d3268e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:47.093545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 664/1309] 10.222.0.20 () {54 vars in 965 bytes} [Fri Sep 9 04:30:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:30:47.099598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd48631-38a8-4634-8d4d-829a4608e270 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:47.100014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd48631-38a8-4634-8d4d-829a4608e270 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:47.100414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd48631-38a8-4634-8d4d-829a4608e270 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:47.182870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0bd48631-38a8-4634-8d4d-829a4608e270 None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:30:47.184207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0bd48631-38a8-4634-8d4d-829a4608e270 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:47.206707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:47.268377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 646/1310] 10.222.0.20 () {58 vars in 1062 bytes} [Fri Sep 9 04:30:47 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:30:47.277415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d01ff-ae97-474e-bc11-471101d3268e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:30:47.279213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c82d01ff-ae97-474e-bc11-471101d3268e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:47.401712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d01ff-ae97-474e-bc11-471101d3268e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:30:47.489057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:30:47.489664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:47.489936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:47.490204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:47.491951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:47.555092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:47.619161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:30:47.620736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d3911c-ec26-48bb-9b0e-0530c9c49cf9 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:30:47.652277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 665/1311] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:30:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:30:50.155004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14603df3-f97e-45c1-bb31-d9dcb08feada None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:50.155551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14603df3-f97e-45c1-bb31-d9dcb08feada None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:50.155551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14603df3-f97e-45c1-bb31-d9dcb08feada None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:50.156882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 647/1312] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:30:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:30:50.162282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b020310d-b980-45f5-bccd-2605781b8152 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:50.162726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b020310d-b980-45f5-bccd-2605781b8152 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:50.162825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b020310d-b980-45f5-bccd-2605781b8152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:50.244581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b020310d-b980-45f5-bccd-2605781b8152 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:30:50.245716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b020310d-b980-45f5-bccd-2605781b8152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:50.342109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 666/1313] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:30:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:30:50.644516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c479a11b-f548-4c80-9427-01f45edc494a None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:50.644810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c479a11b-f548-4c80-9427-01f45edc494a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:50.645090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c479a11b-f548-4c80-9427-01f45edc494a None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:50.646432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 648/1314] 10.222.0.20 () {54 vars in 965 bytes} [Fri Sep 9 04:30:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:30:50.652145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f646fc7b-24b0-40bf-88fc-6f458e0f46f9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:50.652430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f646fc7b-24b0-40bf-88fc-6f458e0f46f9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:50.652709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f646fc7b-24b0-40bf-88fc-6f458e0f46f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:50.664262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:50.728737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f646fc7b-24b0-40bf-88fc-6f458e0f46f9 None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:30:50.729938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f646fc7b-24b0-40bf-88fc-6f458e0f46f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:50.805882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 667/1315] 10.222.0.20 () {58 vars in 1062 bytes} [Fri Sep 9 04:30:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:30:50.815820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c479a11b-f548-4c80-9427-01f45edc494a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:30:50.817368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c479a11b-f548-4c80-9427-01f45edc494a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:50.918512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c479a11b-f548-4c80-9427-01f45edc494a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:30:50.988074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:50.990143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:30:50.990753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:50.991237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:50.991718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:50.993545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:51.101077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:30:51.102505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebebaa0-2c0e-401d-9b0e-89017c2638a4 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:30:51.120390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 649/1316] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:30:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:30:53.795108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be05d22-f77a-4abe-8757-2199ea8acf53 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:53.795656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be05d22-f77a-4abe-8757-2199ea8acf53 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:53.795656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be05d22-f77a-4abe-8757-2199ea8acf53 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:53.797494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 668/1317] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:30:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:30:53.802703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ef630-de86-4752-8249-05e919ff1758 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:53.803150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ef630-de86-4752-8249-05e919ff1758 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:53.803261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ef630-de86-4752-8249-05e919ff1758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:53.867500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5b6ef630-de86-4752-8249-05e919ff1758 None None] MFA Rules not processed for user `9eb2801446984bbe9e1f2c28ec34e108`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:30:53.868686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5b6ef630-de86-4752-8249-05e919ff1758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:53.918795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:53.964365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 650/1318] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:30:53 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:30:54.247031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be05d22-f77a-4abe-8757-2199ea8acf53 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:30:54.248599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3be05d22-f77a-4abe-8757-2199ea8acf53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:54.351501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be05d22-f77a-4abe-8757-2199ea8acf53 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:30:54.354250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:54.424594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:30:54.425187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:54.425462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:54.425730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:54.427321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:54.533401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:30:54.534763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31448d60-e01f-405c-81ea-1e9befea48c3 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:30:54.552665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 669/1319] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:30:54 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:30:57.243084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db8fee-b932-4011-843a-29952f55906c None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:57.243625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db8fee-b932-4011-843a-29952f55906c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:57.243720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db8fee-b932-4011-843a-29952f55906c None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:57.245016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 651/1320] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:30:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:30:57.250159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef366dc-91c9-4eb1-9a67-99209d4adff0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:57.250590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef366dc-91c9-4eb1-9a67-99209d4adff0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:57.250708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef366dc-91c9-4eb1-9a67-99209d4adff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:57.291354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:57.317172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-aef366dc-91c9-4eb1-9a67-99209d4adff0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:30:57.318412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-aef366dc-91c9-4eb1-9a67-99209d4adff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:57.412688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 670/1321] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:30:57 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:30:57.711146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db8fee-b932-4011-843a-29952f55906c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:30:57.712677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-93db8fee-b932-4011-843a-29952f55906c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:57.814099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db8fee-b932-4011-843a-29952f55906c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:30:57.885963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:30:57.886553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:30:57.886833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:30:57.887198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:30:57.888772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:30:57.895583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:30:57.996598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:30:57.997957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6594142a-4de2-4daf-9d1c-4391b8c0354c service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:30:58.015409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 652/1322] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:30:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:00.810303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1036f-02c9-4aeb-aa48-d03c0b3d2450 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:00.810887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1036f-02c9-4aeb-aa48-d03c0b3d2450 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:00.810887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1036f-02c9-4aeb-aa48-d03c0b3d2450 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:00.812279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 671/1323] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:00.817451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41acc8de-5257-40c2-a6f5-7e3e99ca3e0d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:00.817875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41acc8de-5257-40c2-a6f5-7e3e99ca3e0d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:00.817998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41acc8de-5257-40c2-a6f5-7e3e99ca3e0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:00.890078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-41acc8de-5257-40c2-a6f5-7e3e99ca3e0d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:00.891374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-41acc8de-5257-40c2-a6f5-7e3e99ca3e0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:00.953826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:00.994706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 653/1324] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:01.285968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1036f-02c9-4aeb-aa48-d03c0b3d2450 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:01.287621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5ee1036f-02c9-4aeb-aa48-d03c0b3d2450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:01.401217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1036f-02c9-4aeb-aa48-d03c0b3d2450 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:01.414623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:01.481405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:01.482017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:01.482292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:01.482560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:01.484714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:01.603785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:01.605123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0300407-e84d-4b5a-a67f-42570f463d90 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:01.623740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 672/1325] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:03.865897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab4feb-aadd-426d-94a0-9e3f97e431b5 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:03.866474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab4feb-aadd-426d-94a0-9e3f97e431b5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:03.866474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab4feb-aadd-426d-94a0-9e3f97e431b5 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:03.867808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 654/1326] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:03.873148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeaaaa3-5fcf-4391-b94d-8ceaf97ede83 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:03.873584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeaaaa3-5fcf-4391-b94d-8ceaf97ede83 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:03.874262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeaaaa3-5fcf-4391-b94d-8ceaf97ede83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:03.922497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:03.941414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-edeaaaa3-5fcf-4391-b94d-8ceaf97ede83 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:03.942614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-edeaaaa3-5fcf-4391-b94d-8ceaf97ede83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:04.045498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 673/1327] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:04.324522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbab4feb-aadd-426d-94a0-9e3f97e431b5 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:04.326246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dbab4feb-aadd-426d-94a0-9e3f97e431b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:04.428575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbab4feb-aadd-426d-94a0-9e3f97e431b5 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:04.504755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:04.505357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:04.505631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:04.505897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:04.507502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:04.524631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:04.618618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:04.620109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5402b0-f6fb-4c04-942f-432ab5b95186 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:04.639111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 655/1328] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:06.871011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45050b-141e-488a-a01a-e63892e0ed6f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:06.871561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45050b-141e-488a-a01a-e63892e0ed6f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:06.871561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45050b-141e-488a-a01a-e63892e0ed6f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:06.872845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 674/1329] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:06.878575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166f3a4-e11a-419d-a8c9-0df88d729290 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:06.879020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166f3a4-e11a-419d-a8c9-0df88d729290 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:06.879157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166f3a4-e11a-419d-a8c9-0df88d729290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:06.946833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4166f3a4-e11a-419d-a8c9-0df88d729290 None None] MFA Rules not processed for user `9eb2801446984bbe9e1f2c28ec34e108`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:06.948065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4166f3a4-e11a-419d-a8c9-0df88d729290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:07.011632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:07.043508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 656/1330] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:06 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:07.331842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45050b-141e-488a-a01a-e63892e0ed6f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:07.333395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0a45050b-141e-488a-a01a-e63892e0ed6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:07.443618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45050b-141e-488a-a01a-e63892e0ed6f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:07.468058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:07.521996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:07.522599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:07.522873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:07.523173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:07.524745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:07.632524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:07.633944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81fe4b2-fe7f-42fb-97d3-ce1f7cdc80b4 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:07.652085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 675/1331] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:07 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:10.030162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd7868-4ea6-4c7e-9b82-46c06d35386f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:10.030707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd7868-4ea6-4c7e-9b82-46c06d35386f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:10.030803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd7868-4ea6-4c7e-9b82-46c06d35386f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:10.032142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 657/1332] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:10.037336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9faa3ed-d600-4304-9b51-eb4ab1069037 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:10.037784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9faa3ed-d600-4304-9b51-eb4ab1069037 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:10.037875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9faa3ed-d600-4304-9b51-eb4ab1069037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:10.100991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:10.103756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f9faa3ed-d600-4304-9b51-eb4ab1069037 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:10.105840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f9faa3ed-d600-4304-9b51-eb4ab1069037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:10.201541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 676/1333] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:10.486755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bd7868-4ea6-4c7e-9b82-46c06d35386f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:10.488332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-51bd7868-4ea6-4c7e-9b82-46c06d35386f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:10.596698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bd7868-4ea6-4c7e-9b82-46c06d35386f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:10.671550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:10.672152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:10.672426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:10.672692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:10.674286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:10.701322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:10.782038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:10.783427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439f7ee4-35c2-4632-bf6a-e55f17135581 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:10.801386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 658/1334] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:12.989677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75552f7c-cd36-45f5-b4d1-9ac2e0c8ddb9 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:12.990223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75552f7c-cd36-45f5-b4d1-9ac2e0c8ddb9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:12.990223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75552f7c-cd36-45f5-b4d1-9ac2e0c8ddb9 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:12.991619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 677/1335] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:12.996871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8e2bd-8190-4721-9507-52d62d219055 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:12.997250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8e2bd-8190-4721-9507-52d62d219055 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:12.997420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8e2bd-8190-4721-9507-52d62d219055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:13.061638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-1ed8e2bd-8190-4721-9507-52d62d219055 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:13.062833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1ed8e2bd-8190-4721-9507-52d62d219055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:13.134874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:13.158119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 659/1336] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:13.442263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75552f7c-cd36-45f5-b4d1-9ac2e0c8ddb9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:13.443897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-75552f7c-cd36-45f5-b4d1-9ac2e0c8ddb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:13.552620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75552f7c-cd36-45f5-b4d1-9ac2e0c8ddb9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:13.582550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:13.629262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:13.629862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:13.630143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:13.630417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:13.632043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:13.742148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:13.743619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea70247-821f-4117-8ee7-c83a3587b1bf service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:13.761401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 678/1337] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:16.259592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810b96c3-e638-45dc-ad37-c735dc4418b9 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:16.260120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810b96c3-e638-45dc-ad37-c735dc4418b9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:16.260120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810b96c3-e638-45dc-ad37-c735dc4418b9 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:16.261644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 660/1338] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:16.267238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b097c5-8684-4f41-8a1d-7359b9459f90 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:16.267692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b097c5-8684-4f41-8a1d-7359b9459f90 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:16.267803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b097c5-8684-4f41-8a1d-7359b9459f90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:16.332595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-51b097c5-8684-4f41-8a1d-7359b9459f90 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:16.333777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-51b097c5-8684-4f41-8a1d-7359b9459f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:16.341912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:16.430135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 679/1339] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:16 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:16.711258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810b96c3-e638-45dc-ad37-c735dc4418b9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:16.712810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-810b96c3-e638-45dc-ad37-c735dc4418b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:16.813883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810b96c3-e638-45dc-ad37-c735dc4418b9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:16.888143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:16.888680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:16.889520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:16.889795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:16.891673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:16.927070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:16.999700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:17.001139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334f3f93-2344-4581-819a-f3f7d67f9615 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:17.019370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 661/1340] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:16 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:21.079535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455b13f9-6329-495f-82b0-bc697357e1c8 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:21.080110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455b13f9-6329-495f-82b0-bc697357e1c8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:21.080110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455b13f9-6329-495f-82b0-bc697357e1c8 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:21.081769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 680/1341] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:21.086848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46187e-2752-42b0-b5fd-dcfac6bb222c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:21.087296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46187e-2752-42b0-b5fd-dcfac6bb222c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:21.087465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46187e-2752-42b0-b5fd-dcfac6bb222c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:21.151940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-cb46187e-2752-42b0-b5fd-dcfac6bb222c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:21.153136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cb46187e-2752-42b0-b5fd-dcfac6bb222c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:21.238091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:21.248956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 662/1342] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:21.534731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455b13f9-6329-495f-82b0-bc697357e1c8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:21.536298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-455b13f9-6329-495f-82b0-bc697357e1c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:21.640883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455b13f9-6329-495f-82b0-bc697357e1c8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:21.678208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:21.715729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:21.716396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:21.716697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:21.716944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:21.718521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:21.828370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:21.829697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146cf8f8-0dcf-4607-9501-2570e41aad73 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:21.848098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 681/1343] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:24.687451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d7d26-5dd7-4a14-b8c6-4e4ccc6af176 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:24.688001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d7d26-5dd7-4a14-b8c6-4e4ccc6af176 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:24.688093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d7d26-5dd7-4a14-b8c6-4e4ccc6af176 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:24.689411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 663/1344] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:24.694603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e2723b-1903-4e24-b186-2a21829ed734 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:24.695064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e2723b-1903-4e24-b186-2a21829ed734 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:24.695214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e2723b-1903-4e24-b186-2a21829ed734 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:24.760364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b1e2723b-1903-4e24-b186-2a21829ed734 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:24.761602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b1e2723b-1903-4e24-b186-2a21829ed734 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:24.780638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:24.857920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 682/1345] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:24 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:25.135658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d7d26-5dd7-4a14-b8c6-4e4ccc6af176 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:25.137208 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a49d7d26-5dd7-4a14-b8c6-4e4ccc6af176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:25.246079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d7d26-5dd7-4a14-b8c6-4e4ccc6af176 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:25.324036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:25.324632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:25.324907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:25.325183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:25.326780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:25.370507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:25.439048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:25.440503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237e7970-1267-4e33-9b5e-a83e5224f504 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:25.458695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 664/1346] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:25 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:29.606525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd7cea-9792-4043-9b91-2eb91046e3e8 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:29.607120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd7cea-9792-4043-9b91-2eb91046e3e8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:29.607767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd7cea-9792-4043-9b91-2eb91046e3e8 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:29.609018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 683/1347] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:29.614323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7060fb-784f-4902-8a1d-bca96a6c3745 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:29.614729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7060fb-784f-4902-8a1d-bca96a6c3745 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:29.614869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7060fb-784f-4902-8a1d-bca96a6c3745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:29.684356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-cf7060fb-784f-4902-8a1d-bca96a6c3745 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:29.685590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cf7060fb-784f-4902-8a1d-bca96a6c3745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:29.788656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:29.790797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 665/1348] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:30.072773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd7cea-9792-4043-9b91-2eb91046e3e8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:30.074324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c1cd7cea-9792-4043-9b91-2eb91046e3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:30.184078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd7cea-9792-4043-9b91-2eb91046e3e8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:30.234659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:30.260907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:30.261675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:30.261792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:30.262027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:30.264270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:30.376292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:30.377631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3398eba-4855-4742-afb6-c62f5f37516e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:30.396686 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 684/1349] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:35.280301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1af74-4dcc-4aa0-b11d-095cf39e80c3 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:35.280793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1af74-4dcc-4aa0-b11d-095cf39e80c3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:35.280910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1af74-4dcc-4aa0-b11d-095cf39e80c3 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:35.283307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 666/1350] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:31:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:35.287938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65fe37b-4c1d-4941-a9bd-8fb31e165335 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:35.288359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65fe37b-4c1d-4941-a9bd-8fb31e165335 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:35.289115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65fe37b-4c1d-4941-a9bd-8fb31e165335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:35.414968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f65fe37b-4c1d-4941-a9bd-8fb31e165335 None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:35.418089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f65fe37b-4c1d-4941-a9bd-8fb31e165335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:35.484661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:35.585278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 685/1351] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:31:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:39.820128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68668d6f-5efd-4641-aecf-872157dc8715 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:39.821064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68668d6f-5efd-4641-aecf-872157dc8715 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:39.821583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68668d6f-5efd-4641-aecf-872157dc8715 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:39.823514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 667/1352] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:31:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:39.828647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ff1cc8-1978-4d79-848f-0c62285bd08c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:39.829449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ff1cc8-1978-4d79-848f-0c62285bd08c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:39.829944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ff1cc8-1978-4d79-848f-0c62285bd08c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:39.909461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-10ff1cc8-1978-4d79-848f-0c62285bd08c None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:39.911076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-10ff1cc8-1978-4d79-848f-0c62285bd08c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:39.987963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:40.033778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 686/1353] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:31:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:41.285786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845493ba-7661-488c-b088-30778086a9d2 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:41.286326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845493ba-7661-488c-b088-30778086a9d2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:41.286483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845493ba-7661-488c-b088-30778086a9d2 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:41.287851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 668/1354] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:31:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:41.292917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da8cce-3b62-4162-b59f-186ff6932508 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:41.293335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da8cce-3b62-4162-b59f-186ff6932508 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:41.293471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da8cce-3b62-4162-b59f-186ff6932508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:41.371235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-20da8cce-3b62-4162-b59f-186ff6932508 None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:41.372548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-20da8cce-3b62-4162-b59f-186ff6932508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:41.482648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 687/1355] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:31:41 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:45.045054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdf4c25-7269-4e62-8c8c-c2556076f280 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:45.045626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdf4c25-7269-4e62-8c8c-c2556076f280 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:45.045626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdf4c25-7269-4e62-8c8c-c2556076f280 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:45.050963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 669/1356] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:45.053142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bab4c9-7510-4190-9184-6e3c1ae83d62 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:45.054017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bab4c9-7510-4190-9184-6e3c1ae83d62 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:45.055335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bab4c9-7510-4190-9184-6e3c1ae83d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:45.097342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:45.132380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-88bab4c9-7510-4190-9184-6e3c1ae83d62 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:45.133585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-88bab4c9-7510-4190-9184-6e3c1ae83d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:45.242979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 688/1357] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:45.547771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d4700a-cadf-4548-b3b5-b6311323c23b None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:45.548434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d4700a-cadf-4548-b3b5-b6311323c23b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:45.548938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d4700a-cadf-4548-b3b5-b6311323c23b None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:45.550901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 670/1358] 10.222.0.20 () {54 vars in 962 bytes} [Fri Sep 9 04:31:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:45.559044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf1a2b8-1e3a-4a68-90ca-fa6cdb9f46d1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:45.559599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf1a2b8-1e3a-4a68-90ca-fa6cdb9f46d1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:45.560096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf1a2b8-1e3a-4a68-90ca-fa6cdb9f46d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:45.655361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bcf1a2b8-1e3a-4a68-90ca-fa6cdb9f46d1 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:45.656577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bcf1a2b8-1e3a-4a68-90ca-fa6cdb9f46d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:45.707987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:45.780235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 689/1359] 10.222.0.20 () {58 vars in 1059 bytes} [Fri Sep 9 04:31:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:45.787963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d4700a-cadf-4548-b3b5-b6311323c23b None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:45.789683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-11d4700a-cadf-4548-b3b5-b6311323c23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:45.949157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d4700a-cadf-4548-b3b5-b6311323c23b None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:46.033364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:46.076470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:46.077248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:46.077507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:46.077852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:46.079552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:46.175266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:46.177074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caec0f3-3b21-4339-9681-885ad8f3f1cd service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:46.197776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 671/1360] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:48.598848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e7b7f7-c573-4006-870b-7c43e7c4901d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:48.599439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e7b7f7-c573-4006-870b-7c43e7c4901d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:48.599439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e7b7f7-c573-4006-870b-7c43e7c4901d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:48.600760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 690/1361] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:48.606115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c66ed-7145-484c-8f25-e8a0fda97259 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:48.606642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c66ed-7145-484c-8f25-e8a0fda97259 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:48.606742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c66ed-7145-484c-8f25-e8a0fda97259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:48.675161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c72c66ed-7145-484c-8f25-e8a0fda97259 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:48.676371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c72c66ed-7145-484c-8f25-e8a0fda97259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:48.715370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:48.782324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 672/1362] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:49.070700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e7b7f7-c573-4006-870b-7c43e7c4901d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:49.072463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-75e7b7f7-c573-4006-870b-7c43e7c4901d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:49.245455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e7b7f7-c573-4006-870b-7c43e7c4901d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:49.258107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:49.396364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:49.397577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:49.397577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:49.397577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:49.401277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:49.499678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:49.501304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561f6821-595a-4c2d-ac34-7878d49ee202 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:49.520129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 691/1363] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:52.009939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee473e9-8f89-4739-9011-c900ba1b0542 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:52.010507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee473e9-8f89-4739-9011-c900ba1b0542 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:52.010725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee473e9-8f89-4739-9011-c900ba1b0542 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:52.012371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 673/1364] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:52.017910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d01cb0-0dc8-47d6-a700-409db5259518 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:52.018382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d01cb0-0dc8-47d6-a700-409db5259518 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:52.018481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d01cb0-0dc8-47d6-a700-409db5259518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:52.036055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:52.092480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-37d01cb0-0dc8-47d6-a700-409db5259518 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:52.093707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-37d01cb0-0dc8-47d6-a700-409db5259518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:52.194991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 692/1365] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:52.487885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4050ce3-2802-4fbf-9b0d-4ac076e8da63 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:52.488258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4050ce3-2802-4fbf-9b0d-4ac076e8da63 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:52.488556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4050ce3-2802-4fbf-9b0d-4ac076e8da63 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:52.490068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 674/1366] 10.222.0.20 () {54 vars in 962 bytes} [Fri Sep 9 04:31:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:52.495498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684a1140-48a7-4847-8f18-903423812559 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:52.495858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684a1140-48a7-4847-8f18-903423812559 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:52.496229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684a1140-48a7-4847-8f18-903423812559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:52.578163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-684a1140-48a7-4847-8f18-903423812559 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:52.579398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-684a1140-48a7-4847-8f18-903423812559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:52.617307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:52.706914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 693/1367] 10.222.0.20 () {58 vars in 1059 bytes} [Fri Sep 9 04:31:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:52.714609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4050ce3-2802-4fbf-9b0d-4ac076e8da63 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:52.716736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e4050ce3-2802-4fbf-9b0d-4ac076e8da63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:52.871408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4050ce3-2802-4fbf-9b0d-4ac076e8da63 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:52.892474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:53.004004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:53.004625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:53.004910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:53.005184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:53.006794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:53.103790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:53.105406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d000138f-3e50-4863-bb27-8c5b8881f6a8 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:53.125440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 675/1368] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:53.329805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a86ac2-b1ac-41a1-98ee-0b6034b1bc08 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:53.330391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a86ac2-b1ac-41a1-98ee-0b6034b1bc08 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:53.330628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a86ac2-b1ac-41a1-98ee-0b6034b1bc08 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:53.332053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 694/1369] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:53.338046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a992a8-98be-4dec-a78a-61a1ab6c1b1f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:53.338371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a992a8-98be-4dec-a78a-61a1ab6c1b1f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:53.338664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a992a8-98be-4dec-a78a-61a1ab6c1b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:53.377864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:53.424841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b4a992a8-98be-4dec-a78a-61a1ab6c1b1f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:53.425434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b4a992a8-98be-4dec-a78a-61a1ab6c1b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:53.533312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 676/1370] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:53.870785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae184a34-e3be-45dd-8606-57c0ca43727b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:53.870785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae184a34-e3be-45dd-8606-57c0ca43727b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:53.870785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae184a34-e3be-45dd-8606-57c0ca43727b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:53.871891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 695/1371] 10.222.0.20 () {54 vars in 964 bytes} [Fri Sep 9 04:31:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:53.876564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f21246-5ee8-4e06-b239-b37290052688 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:53.876856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f21246-5ee8-4e06-b239-b37290052688 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:53.877134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f21246-5ee8-4e06-b239-b37290052688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:53.959689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-36f21246-5ee8-4e06-b239-b37290052688 None None] MFA Rules not processed for user `a8add98306764181b5ea1e8a1190bbde`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:53.960908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-36f21246-5ee8-4e06-b239-b37290052688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:54.011142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:54.042250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 677/1372] 10.222.0.20 () {58 vars in 1061 bytes} [Fri Sep 9 04:31:53 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:54.050369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae184a34-e3be-45dd-8606-57c0ca43727b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:54.051918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ae184a34-e3be-45dd-8606-57c0ca43727b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:54.164982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae184a34-e3be-45dd-8606-57c0ca43727b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:54.208320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:54.254265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:54.254773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:54.255199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:54.255602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:54.257156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:54.377365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:54.379131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509843f2-1eb8-403b-95a6-1a8c721cf8de service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:54.400411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 696/1373] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:55.413692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a381ffd3-5322-4bb8-b6f1-d268215eda78 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:55.414272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a381ffd3-5322-4bb8-b6f1-d268215eda78 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:55.414272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a381ffd3-5322-4bb8-b6f1-d268215eda78 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:55.415628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 678/1374] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:55.421314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55924a3f-e38c-45c0-bf03-01fc3af9e029 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:55.421759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55924a3f-e38c-45c0-bf03-01fc3af9e029 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:55.422038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55924a3f-e38c-45c0-bf03-01fc3af9e029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:55.488152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-55924a3f-e38c-45c0-bf03-01fc3af9e029 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:55.489361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-55924a3f-e38c-45c0-bf03-01fc3af9e029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:55.510252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:55.586781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 697/1375] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:55.877298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a381ffd3-5322-4bb8-b6f1-d268215eda78 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:55.878871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a381ffd3-5322-4bb8-b6f1-d268215eda78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:56.036205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a381ffd3-5322-4bb8-b6f1-d268215eda78 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:56.088575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:56.151142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:56.151758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:56.152038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:56.152312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:56.154508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:56.252973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:56.254579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0fd9b6-de4b-4d01-aafd-87dc7a00a426 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:56.275791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 679/1376] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:31:58.501714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4d135-a7ab-4101-bf13-3172ff970077 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:58.502278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4d135-a7ab-4101-bf13-3172ff970077 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:58.502278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4d135-a7ab-4101-bf13-3172ff970077 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:58.503991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 698/1377] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:31:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:31:58.509845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a6c5d9-a5e2-4910-a1d8-ebd0a630d809 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:58.510306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a6c5d9-a5e2-4910-a1d8-ebd0a630d809 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:58.510407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a6c5d9-a5e2-4910-a1d8-ebd0a630d809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:58.581252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c2a6c5d9-a5e2-4910-a1d8-ebd0a630d809 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:31:58.582361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c2a6c5d9-a5e2-4910-a1d8-ebd0a630d809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:58.590814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:58.694192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 680/1378] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:31:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:31:58.985107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a4d135-a7ab-4101-bf13-3172ff970077 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:31:58.986602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c1a4d135-a7ab-4101-bf13-3172ff970077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:59.141196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a4d135-a7ab-4101-bf13-3172ff970077 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:31:59.145206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:31:59.268685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:31:59.269313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:31:59.269593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:31:59.269884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:31:59.271524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:31:59.367767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:31:59.369877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0f63b3-8084-4462-8129-9d5c07dc1f0b service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:31:59.389238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 699/1379] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:31:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:01.604591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d5e38c-02f7-48bc-abf8-8d5607302fad None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:01.605489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d5e38c-02f7-48bc-abf8-8d5607302fad None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:01.605489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d5e38c-02f7-48bc-abf8-8d5607302fad None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:01.611551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 681/1380] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:01 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:01.613026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8eacee-d183-4a8e-a7de-678bff1482ed None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:01.613485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8eacee-d183-4a8e-a7de-678bff1482ed None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:01.613649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8eacee-d183-4a8e-a7de-678bff1482ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:01.627325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:01.687071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9f8eacee-d183-4a8e-a7de-678bff1482ed None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:01.688492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9f8eacee-d183-4a8e-a7de-678bff1482ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:01.847122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 700/1381] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:32:02.153543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d5e38c-02f7-48bc-abf8-8d5607302fad None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:02.155148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-62d5e38c-02f7-48bc-abf8-8d5607302fad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:02.297391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:02.308960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d5e38c-02f7-48bc-abf8-8d5607302fad None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:02.441612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:02.442275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:02.442545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:02.442841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:02.444615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:02.544630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:02.546292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67bcc57-31df-44e8-bf18-bf372a587811 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:02.557387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:02.569343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 682/1382] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:05.149945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03aca0e0-e0e3-4a8e-aa9b-4fae81c26f60 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:05.150511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03aca0e0-e0e3-4a8e-aa9b-4fae81c26f60 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:05.150511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03aca0e0-e0e3-4a8e-aa9b-4fae81c26f60 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:05.152395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 701/1383] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:05.158372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b28b877-f715-4c09-b034-c2b1b1df5f63 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:05.158851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b28b877-f715-4c09-b034-c2b1b1df5f63 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:05.158968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b28b877-f715-4c09-b034-c2b1b1df5f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:05.226695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-8b28b877-f715-4c09-b034-c2b1b1df5f63 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:05.227942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8b28b877-f715-4c09-b034-c2b1b1df5f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:05.332156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 683/1384] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:05.623510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03aca0e0-e0e3-4a8e-aa9b-4fae81c26f60 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:05.625506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-03aca0e0-e0e3-4a8e-aa9b-4fae81c26f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:05.726859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:05.790064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03aca0e0-e0e3-4a8e-aa9b-4fae81c26f60 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:05.919758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:05.920575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:05.920858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:05.921265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:05.924759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:05.980492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:06.022853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:06.024506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914c4fbe-3370-42a0-8c8e-27188aa1adec service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:06.042481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 702/1385] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:08.496385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ca4a1-6a15-456f-bd60-52dab864c598 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:08.497000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ca4a1-6a15-456f-bd60-52dab864c598 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:08.497000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ca4a1-6a15-456f-bd60-52dab864c598 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:08.498283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 684/1386] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:08.503688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055e2a8-e287-466d-8f28-761367ce9046 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:08.504160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055e2a8-e287-466d-8f28-761367ce9046 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:08.504272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055e2a8-e287-466d-8f28-761367ce9046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:08.570567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5055e2a8-e287-466d-8f28-761367ce9046 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:08.571724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5055e2a8-e287-466d-8f28-761367ce9046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:08.675167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 703/1387] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:08.970249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893ca4a1-6a15-456f-bd60-52dab864c598 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:08.971851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-893ca4a1-6a15-456f-bd60-52dab864c598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:09.048889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:09.118486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893ca4a1-6a15-456f-bd60-52dab864c598 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:09.230601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:09.231277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:09.231725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:09.232001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:09.233688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:09.278241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:09.331032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:09.332646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6021978-712d-495a-9ebc-36b01439c073 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:09.351466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 685/1388] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:09.566700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d3bb91-176c-4a50-96e1-4287e00fb380 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:09.567305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d3bb91-176c-4a50-96e1-4287e00fb380 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:09.567305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d3bb91-176c-4a50-96e1-4287e00fb380 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:09.568674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 704/1389] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:09.575223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80654aca-975a-4d7e-87bb-de45b48b9b4b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:09.575794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80654aca-975a-4d7e-87bb-de45b48b9b4b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:09.575794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80654aca-975a-4d7e-87bb-de45b48b9b4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:09.642457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-80654aca-975a-4d7e-87bb-de45b48b9b4b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:09.643694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-80654aca-975a-4d7e-87bb-de45b48b9b4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:09.740184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 686/1390] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:10.049516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b3c47-2bde-4763-963e-8f74236d09b0 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:10.049892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b3c47-2bde-4763-963e-8f74236d09b0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:10.050178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b3c47-2bde-4763-963e-8f74236d09b0 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:10.051547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 705/1391] 10.222.0.20 () {54 vars in 966 bytes} [Fri Sep 9 04:32:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:10.056967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d498c116-a9fb-4cf8-97b8-cf2aa04930cd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:10.057318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d498c116-a9fb-4cf8-97b8-cf2aa04930cd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:10.057617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d498c116-a9fb-4cf8-97b8-cf2aa04930cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:10.073921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:10.136158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d498c116-a9fb-4cf8-97b8-cf2aa04930cd None None] MFA Rules not processed for user `fa7372a6f3ba4f3786becc1389147d2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:10.137898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d498c116-a9fb-4cf8-97b8-cf2aa04930cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:10.211027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 687/1392] 10.222.0.20 () {58 vars in 1063 bytes} [Fri Sep 9 04:32:10 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:10.220547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81b3c47-2bde-4763-963e-8f74236d09b0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:10.222135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a81b3c47-2bde-4763-963e-8f74236d09b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:10.263232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:10.345699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81b3c47-2bde-4763-963e-8f74236d09b0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:10.419687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:10.420288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:10.420567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:10.420840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:10.422426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:10.500248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:10.533768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:10.535145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e28fb85-eb2c-4387-a599-5374f9fc36d7 service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:10.553300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 706/1393] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:11.331652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63944cbc-1dfb-4f82-a27e-13e1d6c5a69f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:11.332145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63944cbc-1dfb-4f82-a27e-13e1d6c5a69f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:11.332249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63944cbc-1dfb-4f82-a27e-13e1d6c5a69f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:11.398001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-63944cbc-1dfb-4f82-a27e-13e1d6c5a69f None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:11.399231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-63944cbc-1dfb-4f82-a27e-13e1d6c5a69f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:11.458315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:11.491315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 688/1394] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:11.533054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe98fba-ca75-4b48-89fd-4e23c5a91281 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:11.533561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe98fba-ca75-4b48-89fd-4e23c5a91281 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:11.533669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe98fba-ca75-4b48-89fd-4e23c5a91281 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:11.535051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 707/1395] 10.222.0.20 () {54 vars in 943 bytes} [Fri Sep 9 04:32:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:11.541215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:11.541501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:11.541795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:11.553689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc519a-dc7d-46a3-b3cc-bdda9f7ef872 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:11.554180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc519a-dc7d-46a3-b3cc-bdda9f7ef872 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:11.554629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc519a-dc7d-46a3-b3cc-bdda9f7ef872 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:11.556042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 708/1396] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:32:11.561494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:11.561786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:11.562062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:11.622779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] MFA Rules not processed for user `6a835e652fcc43ab923c9c6ab434d208`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:11.624025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:11.630485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:11.631717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:11.730150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 709/1397] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:32:11.733891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 689/1398] 10.222.0.20 () {58 vars in 1040 bytes} [Fri Sep 9 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:11.743531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:11.744731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:11.783473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:11.848907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8915be7-dcfc-437a-9c73-0bc87f2e0344 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:11.953408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:11.954108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:11.954294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:11.954562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:11.956207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:12.015188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:12.020142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:12.021363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:12.050486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:12.052137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bfe9b8-6084-44bb-b044-6744e2f670a3 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:12.053520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:12.061261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 710/1399] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:32:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 09 04:32:12.127227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539c14b-c1cf-4f6c-8ca1-e92c735fcea7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:12.256685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:12.257300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:12.257578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:12.257844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:12.259494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:12.311826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:12.369200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:12.373364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb0940b-54a5-4bc3-8066-5ddfcf891d4f service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:12.392791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 690/1400] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:32:14.608881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e705837-6e2f-47c3-b8dc-c8dde636ad6b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:14.609931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e705837-6e2f-47c3-b8dc-c8dde636ad6b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:14.610385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e705837-6e2f-47c3-b8dc-c8dde636ad6b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:14.612009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 711/1401] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:14.617676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83618b8d-e55b-4ffa-911b-8052a4d3aa1c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:14.618235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83618b8d-e55b-4ffa-911b-8052a4d3aa1c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:14.618235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83618b8d-e55b-4ffa-911b-8052a4d3aa1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:14.701804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-83618b8d-e55b-4ffa-911b-8052a4d3aa1c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:14.703320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-83618b8d-e55b-4ffa-911b-8052a4d3aa1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:14.833129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 691/1402] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:32:15.152764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e705837-6e2f-47c3-b8dc-c8dde636ad6b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:15.154302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2e705837-6e2f-47c3-b8dc-c8dde636ad6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:15.367323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e705837-6e2f-47c3-b8dc-c8dde636ad6b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:15.434954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:15.519830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:15.520445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:15.520728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:15.521006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:15.522634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:15.647583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:15.649308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60756e31-5d60-4e23-ac5d-24888d4dd9a4 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:15.674768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 712/1403] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:17.952881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82497f2-e16f-43a7-9446-3960cd09e5bb None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:17.953455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82497f2-e16f-43a7-9446-3960cd09e5bb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:17.953455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82497f2-e16f-43a7-9446-3960cd09e5bb None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:17.960599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa1b07-13c5-4300-bd84-90d21d4779cd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:17.961087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa1b07-13c5-4300-bd84-90d21d4779cd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:17.961182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa1b07-13c5-4300-bd84-90d21d4779cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:17.963032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 692/1404] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:18.044835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e0aa1b07-13c5-4300-bd84-90d21d4779cd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:18.046229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e0aa1b07-13c5-4300-bd84-90d21d4779cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:18.049438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:18.202318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 713/1405] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:32:18.483369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82497f2-e16f-43a7-9446-3960cd09e5bb None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:18.484937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c82497f2-e16f-43a7-9446-3960cd09e5bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:18.520404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:18.682946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82497f2-e16f-43a7-9446-3960cd09e5bb None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:18.799910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:18.830579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:18.832051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:18.832398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:18.833327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:18.835215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:18.946832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:18.948492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70f4d95-fa1e-438f-98e6-80289d568274 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:18.972499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 693/1406] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:21.232894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5966a3-7514-45bf-8882-e56be44c7304 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:21.233427 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5966a3-7514-45bf-8882-e56be44c7304 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:21.233427 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5966a3-7514-45bf-8882-e56be44c7304 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:21.238795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 714/1407] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:32:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:21.240416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f2e07e-34e5-436f-b1ef-8fb535cd6622 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:21.240854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f2e07e-34e5-436f-b1ef-8fb535cd6622 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:21.241026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f2e07e-34e5-436f-b1ef-8fb535cd6622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:21.321281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-32f2e07e-34e5-436f-b1ef-8fb535cd6622 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:21.322510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-32f2e07e-34e5-436f-b1ef-8fb535cd6622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:21.388651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:21.447371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 694/1408] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:32:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:21.739190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5966a3-7514-45bf-8882-e56be44c7304 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:21.740817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cd5966a3-7514-45bf-8882-e56be44c7304 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:21.908747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:21.921184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5966a3-7514-45bf-8882-e56be44c7304 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:22.060457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:22.061268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:22.061552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:22.061830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:22.063586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:22.170870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:22.173664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8302cb0-6984-41e9-8358-bfe0223fc38b service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:22.184551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:22.199776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 715/1409] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:31.440152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=19 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:32.196756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:36.534213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e159c125-d0d5-4aec-a8ef-e4f78fb7618e None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:36.534763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e159c125-d0d5-4aec-a8ef-e4f78fb7618e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:36.534763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e159c125-d0d5-4aec-a8ef-e4f78fb7618e None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:36.536092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 695/1410] 10.222.0.20 () {54 vars in 918 bytes} [Fri Sep 9 04:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:36.541439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4226ce-6187-46dd-9d66-b603a69d08ba None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:36.541872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4226ce-6187-46dd-9d66-b603a69d08ba None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:36.541980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4226ce-6187-46dd-9d66-b603a69d08ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:36.618283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8a4226ce-6187-46dd-9d66-b603a69d08ba None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:36.619500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8a4226ce-6187-46dd-9d66-b603a69d08ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:36.726241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 716/1411] 10.222.0.20 () {58 vars in 1015 bytes} [Fri Sep 9 04:32:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:36.741323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac07120-712d-417b-ad67-5b6c0a7c51e9 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:36.741609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac07120-712d-417b-ad67-5b6c0a7c51e9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:36.741885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac07120-712d-417b-ad67-5b6c0a7c51e9 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:36.743542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 696/1412] 10.222.0.20 () {54 vars in 918 bytes} [Fri Sep 9 04:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:36.748696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e4313a-a720-4cd0-96c8-992bbfa1d8c3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:36.748986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e4313a-a720-4cd0-96c8-992bbfa1d8c3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:36.749264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e4313a-a720-4cd0-96c8-992bbfa1d8c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:36.820845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:36.823693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b5e4313a-a720-4cd0-96c8-992bbfa1d8c3 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:36.825438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b5e4313a-a720-4cd0-96c8-992bbfa1d8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:36.943288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 717/1413] 10.222.0.20 () {58 vars in 1015 bytes} [Fri Sep 9 04:32:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:39.522034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d561cf2-22ba-485c-ac45-1094edd5c46a None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:39.522579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d561cf2-22ba-485c-ac45-1094edd5c46a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:39.522579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d561cf2-22ba-485c-ac45-1094edd5c46a None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:39.523868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 697/1414] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:32:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:39.529300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f05c13-709c-4c2d-8865-0d194d54851f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:39.529761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f05c13-709c-4c2d-8865-0d194d54851f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:39.529880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f05c13-709c-4c2d-8865-0d194d54851f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:39.595919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-98f05c13-709c-4c2d-8865-0d194d54851f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:39.597119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-98f05c13-709c-4c2d-8865-0d194d54851f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:39.651464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:39.693660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 718/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:32:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:39.979794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d561cf2-22ba-485c-ac45-1094edd5c46a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:39.981910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4d561cf2-22ba-485c-ac45-1094edd5c46a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:40.125302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d561cf2-22ba-485c-ac45-1094edd5c46a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:40.197161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:40.238541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:40.239101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:40.239405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:40.239673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:40.241306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:40.329668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:40.331333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5599d3-d7b9-43b4-837e-5d3dc49b6855 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:40.349251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 698/1416] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:41.494515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e844276-f5d5-43d0-803d-c0caf8c1c486 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:41.495123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e844276-f5d5-43d0-803d-c0caf8c1c486 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:41.495123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e844276-f5d5-43d0-803d-c0caf8c1c486 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:41.496941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 719/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:41.503064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e50ca-7d16-42fb-be64-4ad29d98f61c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:41.503514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e50ca-7d16-42fb-be64-4ad29d98f61c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:41.503606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e50ca-7d16-42fb-be64-4ad29d98f61c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:41.578465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-cc8e50ca-7d16-42fb-be64-4ad29d98f61c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:41.579625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cc8e50ca-7d16-42fb-be64-4ad29d98f61c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:41.606810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:41.675331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 699/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:41.829543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e844276-f5d5-43d0-803d-c0caf8c1c486 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:41.831006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5e844276-f5d5-43d0-803d-c0caf8c1c486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:41.977405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e844276-f5d5-43d0-803d-c0caf8c1c486 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:42.017514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:42.091444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:42.091975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:42.092253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:42.092523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:42.094111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:42.182577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:42.184689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6839c59-b9e2-4f89-a901-86faee041eb0 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:42.202646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 720/1419] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:43.329153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cab495-c326-401d-ad69-555dbb05cc6c None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:43.329708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cab495-c326-401d-ad69-555dbb05cc6c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:43.329708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cab495-c326-401d-ad69-555dbb05cc6c None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:43.331108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 700/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:43.338357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9695a57-52b5-4be7-bc47-f563a5b60aa2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:43.338792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9695a57-52b5-4be7-bc47-f563a5b60aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:43.338885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9695a57-52b5-4be7-bc47-f563a5b60aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:43.393131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:43.415017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f9695a57-52b5-4be7-bc47-f563a5b60aa2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:43.416155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f9695a57-52b5-4be7-bc47-f563a5b60aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:43.511865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 721/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:43.664049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cab495-c326-401d-ad69-555dbb05cc6c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:43.665628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-46cab495-c326-401d-ad69-555dbb05cc6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:43.807659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cab495-c326-401d-ad69-555dbb05cc6c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:43.819949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:43.917654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:43.918174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:43.918448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:43.918715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:43.920328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:44.007485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:44.009095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab31ec1-50a2-4d68-b2e4-f370b22fde60 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:44.027670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 701/1422] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:45.140017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114aa37-7c62-438c-b600-db24b45d5d33 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:45.140565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114aa37-7c62-438c-b600-db24b45d5d33 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:45.140565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114aa37-7c62-438c-b600-db24b45d5d33 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:45.141835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 722/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:45.148106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfef57cb-0d7b-4acf-b0f2-959ffb262e75 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:45.148532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfef57cb-0d7b-4acf-b0f2-959ffb262e75 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:45.148643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfef57cb-0d7b-4acf-b0f2-959ffb262e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:45.168896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:45.224376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-dfef57cb-0d7b-4acf-b0f2-959ffb262e75 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:45.225584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dfef57cb-0d7b-4acf-b0f2-959ffb262e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:45.323383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 702/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:32:45.479907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8114aa37-7c62-438c-b600-db24b45d5d33 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:45.481456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8114aa37-7c62-438c-b600-db24b45d5d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:45.590975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:45.628276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8114aa37-7c62-438c-b600-db24b45d5d33 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:45.739469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:45.740074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:45.740346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:45.740611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:45.742273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:45.818982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:45.832927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:45.834536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9425ae-35f5-4177-8545-8c332729ad21 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:45.852391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 723/1425] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:46.982387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651907a9-60ec-4539-988d-dc615371ee37 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:46.982989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651907a9-60ec-4539-988d-dc615371ee37 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:46.982989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651907a9-60ec-4539-988d-dc615371ee37 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:46.984235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 703/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:46.990635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa7722-d51b-422a-9421-0dfcebe84d7c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:46.991088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa7722-d51b-422a-9421-0dfcebe84d7c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:46.991194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa7722-d51b-422a-9421-0dfcebe84d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:47.067244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-10aa7722-d51b-422a-9421-0dfcebe84d7c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:47.068430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-10aa7722-d51b-422a-9421-0dfcebe84d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:47.162804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 724/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:47.312697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651907a9-60ec-4539-988d-dc615371ee37 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:47.314234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-651907a9-60ec-4539-988d-dc615371ee37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:47.396651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:47.458378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651907a9-60ec-4539-988d-dc615371ee37 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:47.569279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:47.569802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:47.570083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:47.570359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:47.572030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:47.620760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:47.661256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:47.662852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457de4c-5308-4230-aba7-c7f0b42f8f79 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:47.681003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 704/1428] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:48.752933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2c8b42-fac7-49f9-9fdf-b6785999c3b8 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:48.753514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2c8b42-fac7-49f9-9fdf-b6785999c3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:48.753514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2c8b42-fac7-49f9-9fdf-b6785999c3b8 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:48.754820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 725/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:48.760910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d4665a-6f41-436c-9332-45fcd520ec74 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:48.761325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d4665a-6f41-436c-9332-45fcd520ec74 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:48.761483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d4665a-6f41-436c-9332-45fcd520ec74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:48.836377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-40d4665a-6f41-436c-9332-45fcd520ec74 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:48.837600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-40d4665a-6f41-436c-9332-45fcd520ec74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:48.934937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 705/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:49.083635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2c8b42-fac7-49f9-9fdf-b6785999c3b8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:49.085101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cb2c8b42-fac7-49f9-9fdf-b6785999c3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:49.134722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:49.231049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2c8b42-fac7-49f9-9fdf-b6785999c3b8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:49.343384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:49.343913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:49.344188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:49.344456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:49.346035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:49.352808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:49.434504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:49.436140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261e4d3b-4349-4546-b330-d11bc1c7d3fa service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:49.454141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 726/1431] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:50.508635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea5706-8bbb-44c4-b0ef-5fc37b15c206 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:50.509215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea5706-8bbb-44c4-b0ef-5fc37b15c206 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:50.509215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea5706-8bbb-44c4-b0ef-5fc37b15c206 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:50.510525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 706/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:50.517617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccee4231-e83e-41b2-98f3-6a64d31637f8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:50.518053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccee4231-e83e-41b2-98f3-6a64d31637f8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:50.518155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccee4231-e83e-41b2-98f3-6a64d31637f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:50.593879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ccee4231-e83e-41b2-98f3-6a64d31637f8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:50.595537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ccee4231-e83e-41b2-98f3-6a64d31637f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:50.663579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:50.694203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 727/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:50.845005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea5706-8bbb-44c4-b0ef-5fc37b15c206 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:50.846469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cbea5706-8bbb-44c4-b0ef-5fc37b15c206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:50.864312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:50.991474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea5706-8bbb-44c4-b0ef-5fc37b15c206 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:51.078125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:51.109117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:51.109659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:51.109944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:51.110222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:51.111878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:51.200909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:51.202554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aa83a2-a6b4-492b-8361-b145f487997b service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:51.220707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 707/1434] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:52.264564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aac8f96-6342-4241-9817-adc14ec5b067 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:52.264981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aac8f96-6342-4241-9817-adc14ec5b067 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:52.265123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aac8f96-6342-4241-9817-adc14ec5b067 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:52.266543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 728/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:52.272808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99d1e83-dbc7-4e42-a7d4-8db7d46d3bb1 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:52.273217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99d1e83-dbc7-4e42-a7d4-8db7d46d3bb1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:52.273317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99d1e83-dbc7-4e42-a7d4-8db7d46d3bb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:52.349825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f99d1e83-dbc7-4e42-a7d4-8db7d46d3bb1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:52.351072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f99d1e83-dbc7-4e42-a7d4-8db7d46d3bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:52.391821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:52.445849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 708/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:52.593903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aac8f96-6342-4241-9817-adc14ec5b067 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:52.595391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5aac8f96-6342-4241-9817-adc14ec5b067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:52.741965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aac8f96-6342-4241-9817-adc14ec5b067 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:52.795613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:52.854050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:52.854576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:52.854848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:52.855720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:52.857306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:52.946241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:52.947860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596fc582-abb1-4009-ac34-50c1cbaa2149 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:52.965826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 729/1437] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:54.011998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee39d4a0-da7a-4f67-b524-a9197d992692 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:54.012555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee39d4a0-da7a-4f67-b524-a9197d992692 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:54.012555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee39d4a0-da7a-4f67-b524-a9197d992692 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:54.013890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 709/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:32:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:54.020136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7fbb80-cf88-4ed1-9525-7e1b328d3f08 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:54.020534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7fbb80-cf88-4ed1-9525-7e1b328d3f08 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:54.020695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7fbb80-cf88-4ed1-9525-7e1b328d3f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:54.096129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2a7fbb80-cf88-4ed1-9525-7e1b328d3f08 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:54.097314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2a7fbb80-cf88-4ed1-9525-7e1b328d3f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:54.099332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:54.314909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 730/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:32:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:54.464198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee39d4a0-da7a-4f67-b524-a9197d992692 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:54.466245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ee39d4a0-da7a-4f67-b524-a9197d992692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:54.610803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee39d4a0-da7a-4f67-b524-a9197d992692 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:54.637536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:54.724003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:54.724613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:54.724892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:54.725163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:54.726775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:54.814843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:54.816504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981df07b-ab3b-476f-ac97-a96be2c505ac service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:54.834711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 710/1440] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:56.889601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acec99f-62c5-484c-936e-2cf3211c6685 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:56.890136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acec99f-62c5-484c-936e-2cf3211c6685 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:56.890136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acec99f-62c5-484c-936e-2cf3211c6685 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:56.891488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 731/1441] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:32:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:56.896704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81a7e5b-9979-496d-a300-5e4a626211a4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:56.897150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81a7e5b-9979-496d-a300-5e4a626211a4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:56.897256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81a7e5b-9979-496d-a300-5e4a626211a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:56.938084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:56.965100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a81a7e5b-9979-496d-a300-5e4a626211a4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:32:56.966419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a81a7e5b-9979-496d-a300-5e4a626211a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:57.064186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 711/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:32:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:32:57.353836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acec99f-62c5-484c-936e-2cf3211c6685 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:32:57.355418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5acec99f-62c5-484c-936e-2cf3211c6685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:57.457096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acec99f-62c5-484c-936e-2cf3211c6685 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:32:57.488122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:32:57.529936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:32:57.530468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:57.530746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:57.531049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:57.532635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:32:57.641313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:32:57.642733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4b0446-145b-4617-9351-7ccb30ef77c1 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:32:57.660389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 732/1443] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:32:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:32:59.957211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc158c6d-a5f2-4efd-9c54-1e995cba3a55 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:59.957783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc158c6d-a5f2-4efd-9c54-1e995cba3a55 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:59.957783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc158c6d-a5f2-4efd-9c54-1e995cba3a55 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:32:59.959137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 712/1444] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:32:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:32:59.964436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d71ea6f-cba0-4ff8-9e8f-704a6f32e04e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:32:59.964827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d71ea6f-cba0-4ff8-9e8f-704a6f32e04e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:32:59.964987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d71ea6f-cba0-4ff8-9e8f-704a6f32e04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:00.033055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9d71ea6f-cba0-4ff8-9e8f-704a6f32e04e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:00.034293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9d71ea6f-cba0-4ff8-9e8f-704a6f32e04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:00.047466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:00.131041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 733/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:32:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:00.411115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc158c6d-a5f2-4efd-9c54-1e995cba3a55 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:00.412699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bc158c6d-a5f2-4efd-9c54-1e995cba3a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:00.515068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc158c6d-a5f2-4efd-9c54-1e995cba3a55 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:00.527597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:00.588120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:00.588721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:00.588999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:00.589270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:00.590864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:00.699911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:00.701345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d188ab-2fcd-460f-8441-7b647dcac551 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:00.719682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 713/1446] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:03.175226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb0aa0d-a505-408e-8aa3-938fd3bedfdb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:03.175777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb0aa0d-a505-408e-8aa3-938fd3bedfdb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:03.175777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb0aa0d-a505-408e-8aa3-938fd3bedfdb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:03.177129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 734/1447] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:03.182581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a2752-0320-49f7-9519-8205ed6278c1 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:03.183358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a2752-0320-49f7-9519-8205ed6278c1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:03.183469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a2752-0320-49f7-9519-8205ed6278c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:03.231797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:03.249936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ce7a2752-0320-49f7-9519-8205ed6278c1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:03.251181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ce7a2752-0320-49f7-9519-8205ed6278c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:03.347518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 714/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:03.634228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb0aa0d-a505-408e-8aa3-938fd3bedfdb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:03.635810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6fb0aa0d-a505-408e-8aa3-938fd3bedfdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:03.737986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb0aa0d-a505-408e-8aa3-938fd3bedfdb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:03.778335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:03.811652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:03.812249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:03.812531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:03.812796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:03.814376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:03.922691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:03.924170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4b2165-339e-49ca-8843-d140466660f8 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:03.942485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 735/1449] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:06.411388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784fdf57-b3be-4b05-894a-47f581bb2bd4 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:06.411951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784fdf57-b3be-4b05-894a-47f581bb2bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:06.411951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784fdf57-b3be-4b05-894a-47f581bb2bd4 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:06.413258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 715/1450] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:06.419062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1779b-528d-4a54-95a5-faf005dc68ea None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:06.419544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1779b-528d-4a54-95a5-faf005dc68ea None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:06.419669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1779b-528d-4a54-95a5-faf005dc68ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:06.485052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d7c1779b-528d-4a54-95a5-faf005dc68ea None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:06.486255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d7c1779b-528d-4a54-95a5-faf005dc68ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:06.506312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:06.582996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 736/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:06.868473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784fdf57-b3be-4b05-894a-47f581bb2bd4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:06.870054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-784fdf57-b3be-4b05-894a-47f581bb2bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:06.971910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784fdf57-b3be-4b05-894a-47f581bb2bd4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:06.991287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:07.044865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:07.045463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:07.045741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:07.046012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:07.047643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:07.155290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:07.156734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45ac5ed-e216-43ef-bc2c-243ed41af07c service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:07.174189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 716/1452] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:09.442685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a76540-57fe-40d1-8720-af19a1e43559 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:09.443292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a76540-57fe-40d1-8720-af19a1e43559 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:09.443292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a76540-57fe-40d1-8720-af19a1e43559 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:09.445512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 737/1453] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:09.451724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a117e610-f15c-45b3-87b6-d9de8c4d9364 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:09.452139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a117e610-f15c-45b3-87b6-d9de8c4d9364 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:09.452307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a117e610-f15c-45b3-87b6-d9de8c4d9364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:09.515260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:09.517925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a117e610-f15c-45b3-87b6-d9de8c4d9364 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:09.519714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a117e610-f15c-45b3-87b6-d9de8c4d9364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:09.614669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 717/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:09.898692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a76540-57fe-40d1-8720-af19a1e43559 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:09.900278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-00a76540-57fe-40d1-8720-af19a1e43559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:10.003098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a76540-57fe-40d1-8720-af19a1e43559 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:10.051749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:10.077346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:10.077941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:10.078215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:10.078483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:10.080086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:10.188372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:10.189790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad918fd4-69a0-4c89-8e00-1d2402731882 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:10.207687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 738/1455] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:12.547063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb58a0d-9f54-4035-b8f5-79ec94429351 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:12.547617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb58a0d-9f54-4035-b8f5-79ec94429351 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:12.547617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb58a0d-9f54-4035-b8f5-79ec94429351 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:12.548976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 718/1456] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:12.554245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b1a823-c24d-42ff-8552-a30ab5c61dc6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:12.554687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b1a823-c24d-42ff-8552-a30ab5c61dc6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:12.554831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b1a823-c24d-42ff-8552-a30ab5c61dc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:12.620764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-53b1a823-c24d-42ff-8552-a30ab5c61dc6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:12.621966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-53b1a823-c24d-42ff-8552-a30ab5c61dc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:12.651314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:12.719036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 739/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:13.004231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb58a0d-9f54-4035-b8f5-79ec94429351 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:13.005811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5fb58a0d-9f54-4035-b8f5-79ec94429351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:13.109750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb58a0d-9f54-4035-b8f5-79ec94429351 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:13.138407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:13.185383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:13.185992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:13.186270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:13.186544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:13.188310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:13.297367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:13.298804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f87de8-8db2-4e60-a40f-b661a7ae106d service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:13.317239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 719/1458] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:15.509630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d91dac-f58e-4f32-9310-b45d6c59ced2 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:15.510198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d91dac-f58e-4f32-9310-b45d6c59ced2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:15.510198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d91dac-f58e-4f32-9310-b45d6c59ced2 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:15.512191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 740/1459] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:15.517670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21e2b6-f225-4b39-8d79-38ccbbc40f97 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:15.518094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21e2b6-f225-4b39-8d79-38ccbbc40f97 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:15.518204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21e2b6-f225-4b39-8d79-38ccbbc40f97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:15.583954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ff21e2b6-f225-4b39-8d79-38ccbbc40f97 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:15.585182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ff21e2b6-f225-4b39-8d79-38ccbbc40f97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:15.593650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:15.682295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 720/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:15.967391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d91dac-f58e-4f32-9310-b45d6c59ced2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:15.968913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e7d91dac-f58e-4f32-9310-b45d6c59ced2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:16.072409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d91dac-f58e-4f32-9310-b45d6c59ced2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:16.127487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:16.145003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:16.145603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:16.145878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:16.146147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:16.147756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:16.254704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:16.256104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c176ffa1-95e6-419b-bff5-764c71cdd23c service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:16.274479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 741/1461] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:18.480347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b1996-1fb3-4a7d-8a6b-c7a7512dfd20 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:18.480948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b1996-1fb3-4a7d-8a6b-c7a7512dfd20 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:18.480948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b1996-1fb3-4a7d-8a6b-c7a7512dfd20 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:18.482240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 721/1462] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:18.487623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c761476-058f-4163-8e1c-c65fc429f19a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:18.488079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c761476-058f-4163-8e1c-c65fc429f19a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:18.488188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c761476-058f-4163-8e1c-c65fc429f19a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:18.553255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4c761476-058f-4163-8e1c-c65fc429f19a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:18.554369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c761476-058f-4163-8e1c-c65fc429f19a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:18.592608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:18.650846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 742/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:18.930460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479b1996-1fb3-4a7d-8a6b-c7a7512dfd20 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:18.932039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-479b1996-1fb3-4a7d-8a6b-c7a7512dfd20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:19.034482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479b1996-1fb3-4a7d-8a6b-c7a7512dfd20 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:19.073101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:19.109366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:19.109965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:19.110240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:19.110506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:19.112132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:19.218456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:19.219838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38819b99-8279-47cc-8166-309ba6798e2b service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:19.237220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 722/1464] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:21.388830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa524e9d-0d0a-4ab5-9e54-4c85bea00deb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:21.389368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa524e9d-0d0a-4ab5-9e54-4c85bea00deb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:21.389368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa524e9d-0d0a-4ab5-9e54-4c85bea00deb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:21.390640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 743/1465] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:21.395917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efd206-cbb8-463c-b056-df83bdbc8e09 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:21.396339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efd206-cbb8-463c-b056-df83bdbc8e09 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:21.396473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efd206-cbb8-463c-b056-df83bdbc8e09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:21.462402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-67efd206-cbb8-463c-b056-df83bdbc8e09 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:21.463629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-67efd206-cbb8-463c-b056-df83bdbc8e09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:21.482429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:21.560529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 723/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:21.846332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa524e9d-0d0a-4ab5-9e54-4c85bea00deb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:21.847938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-aa524e9d-0d0a-4ab5-9e54-4c85bea00deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:21.952201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa524e9d-0d0a-4ab5-9e54-4c85bea00deb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:22.019463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:22.026398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:22.027534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:22.027812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:22.028082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:22.029659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:22.137080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:22.138471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c560da-9754-4b45-902c-d3324e4291ab service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:22.156419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 744/1467] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:24.343402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126a2468-763d-46ba-b60a-bf1577f3d64e None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:24.343929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126a2468-763d-46ba-b60a-bf1577f3d64e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:24.343929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126a2468-763d-46ba-b60a-bf1577f3d64e None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:24.345294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 724/1468] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:33:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:24.350510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50b153-672c-41a3-bc34-05bfae2c189b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:24.350968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50b153-672c-41a3-bc34-05bfae2c189b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:24.351102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50b153-672c-41a3-bc34-05bfae2c189b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:24.417838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6c50b153-672c-41a3-bc34-05bfae2c189b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:24.419090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6c50b153-672c-41a3-bc34-05bfae2c189b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:24.467170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:24.515260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 745/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:33:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:24.805187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126a2468-763d-46ba-b60a-bf1577f3d64e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:24.806734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-126a2468-763d-46ba-b60a-bf1577f3d64e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:24.909534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126a2468-763d-46ba-b60a-bf1577f3d64e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:24.957854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:24.984167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:24.984775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:24.985083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:24.985355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:24.986976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:25.099291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:25.100664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29ac635-e008-4404-9661-1a602cc2aa2a service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:25.118954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 725/1470] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:34.513728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=22 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:35.116724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=66 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:45.042771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deecddbb-3afe-4bdc-ad7c-8b8faf6cff4d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:45.043286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deecddbb-3afe-4bdc-ad7c-8b8faf6cff4d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:45.043382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deecddbb-3afe-4bdc-ad7c-8b8faf6cff4d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:45.044724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 746/1471] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:33:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:45.049889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c408d29-01b4-4fee-bf4b-f19c868c51fd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:45.050346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c408d29-01b4-4fee-bf4b-f19c868c51fd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:45.050456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c408d29-01b4-4fee-bf4b-f19c868c51fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:45.116380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4c408d29-01b4-4fee-bf4b-f19c868c51fd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:45.117512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4c408d29-01b4-4fee-bf4b-f19c868c51fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:45.212074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 726/1472] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:45.508378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df94fc-ec53-4f45-8ac9-dc3f8a52b2eb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:45.508663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df94fc-ec53-4f45-8ac9-dc3f8a52b2eb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:45.508937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df94fc-ec53-4f45-8ac9-dc3f8a52b2eb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:45.510254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 747/1473] 10.222.0.20 () {54 vars in 966 bytes} [Fri Sep 9 04:33:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:45.515582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ff26f-d27c-47d4-9ccc-c7947fe85155 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:45.515869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ff26f-d27c-47d4-9ccc-c7947fe85155 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:45.516165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ff26f-d27c-47d4-9ccc-c7947fe85155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:45.591017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-121ff26f-d27c-47d4-9ccc-c7947fe85155 None None] MFA Rules not processed for user `fa7372a6f3ba4f3786becc1389147d2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:45.592224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-121ff26f-d27c-47d4-9ccc-c7947fe85155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:45.617649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:45.673789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 727/1474] 10.222.0.20 () {58 vars in 1063 bytes} [Fri Sep 9 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:45.681590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df94fc-ec53-4f45-8ac9-dc3f8a52b2eb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:45.683996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-60df94fc-ec53-4f45-8ac9-dc3f8a52b2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:45.787945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df94fc-ec53-4f45-8ac9-dc3f8a52b2eb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:45.862182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:45.862781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:45.863087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:45.863362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:45.864939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:45.919816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:45.974228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:45.975621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5986d7-55d9-4e13-b81e-9b0acc14d54e service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:45.993513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 748/1475] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:48.220221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc96b99-e399-4919-98e6-149c554fda66 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:48.220830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc96b99-e399-4919-98e6-149c554fda66 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:48.220830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc96b99-e399-4919-98e6-149c554fda66 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:48.222148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 728/1476] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:33:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:33:48.227435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d30bd-1d67-4217-9f4a-6910f39ce53f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:48.227848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d30bd-1d67-4217-9f4a-6910f39ce53f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:48.227988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d30bd-1d67-4217-9f4a-6910f39ce53f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:48.297078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-162d30bd-1d67-4217-9f4a-6910f39ce53f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:48.298325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-162d30bd-1d67-4217-9f4a-6910f39ce53f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:48.396654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 749/1477] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:33:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:48.670071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc96b99-e399-4919-98e6-149c554fda66 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:48.671715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8fc96b99-e399-4919-98e6-149c554fda66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:48.795057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc96b99-e399-4919-98e6-149c554fda66 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:48.844545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:48.887931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:48.888541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:48.888821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:48.889096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:48.889740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:48.891569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:33:48.891569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:33:48.891569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:33:48.892718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49cddc2-8016-4793-becd-ea3a1fa263fa admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:48.912058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 729/1478] 10.222.0.20 () {56 vars in 1108 bytes} [Fri Sep 9 04:33:48 2022] GET /identity/v3/projects => generated 2553 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:33:51.070361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521db1ef-185a-498f-bfe3-d2b4b628c42e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:51.071167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521db1ef-185a-498f-bfe3-d2b4b628c42e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:51.071167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521db1ef-185a-498f-bfe3-d2b4b628c42e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:51.072320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 750/1479] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:33:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:51.077774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57664f6-0f7c-4083-8f56-0064172e7c94 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:51.078193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57664f6-0f7c-4083-8f56-0064172e7c94 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:51.078328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57664f6-0f7c-4083-8f56-0064172e7c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:51.145547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c57664f6-0f7c-4083-8f56-0064172e7c94 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:51.146757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c57664f6-0f7c-4083-8f56-0064172e7c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:51.242919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 730/1480] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:33:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:51.539779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521db1ef-185a-498f-bfe3-d2b4b628c42e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:51.541405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-521db1ef-185a-498f-bfe3-d2b4b628c42e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:51.551843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:51.687621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521db1ef-185a-498f-bfe3-d2b4b628c42e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:51.765207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:51.799432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:51.799956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:51.800231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:51.800494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:51.802071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:51.890449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:51.892045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5037786a-83ae-4aa6-b9f2-8215d60eab09 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:51.910068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 751/1481] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:54.239937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b5a17-23a6-4a46-ab2a-1e12f7eb1801 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:54.240507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b5a17-23a6-4a46-ab2a-1e12f7eb1801 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:54.240507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b5a17-23a6-4a46-ab2a-1e12f7eb1801 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:54.241806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 731/1482] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:33:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:54.247202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418f221-3e58-40ef-9b60-8f5be17998d8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:54.247610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418f221-3e58-40ef-9b60-8f5be17998d8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:54.247744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418f221-3e58-40ef-9b60-8f5be17998d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:54.313172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3418f221-3e58-40ef-9b60-8f5be17998d8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:54.314306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3418f221-3e58-40ef-9b60-8f5be17998d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:54.360403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:54.410143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 752/1483] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:33:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:54.696662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b5a17-23a6-4a46-ab2a-1e12f7eb1801 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:54.698280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1b2b5a17-23a6-4a46-ab2a-1e12f7eb1801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:54.722041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:54.802200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b5a17-23a6-4a46-ab2a-1e12f7eb1801 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:54.992855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:54.993468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:54.993937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:54.994202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:54.996396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:55.070381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:55.106180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:55.107654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaff2f2-a349-4a12-940c-e616c6551c6a service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:55.125826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 732/1484] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:33:57.311977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39f9594-b5db-4fe8-8317-d889da79a4f6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:57.312520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39f9594-b5db-4fe8-8317-d889da79a4f6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:57.312520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39f9594-b5db-4fe8-8317-d889da79a4f6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:57.313861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 753/1485] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:33:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:33:57.319166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5f47f8-0330-4ada-bbd1-c998d82f5f7e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:57.319580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5f47f8-0330-4ada-bbd1-c998d82f5f7e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:57.319710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5f47f8-0330-4ada-bbd1-c998d82f5f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:57.384602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3e5f47f8-0330-4ada-bbd1-c998d82f5f7e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:33:57.385807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3e5f47f8-0330-4ada-bbd1-c998d82f5f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:57.478451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 733/1486] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:33:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:33:57.777570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39f9594-b5db-4fe8-8317-d889da79a4f6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:33:57.779151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c39f9594-b5db-4fe8-8317-d889da79a4f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:57.881186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39f9594-b5db-4fe8-8317-d889da79a4f6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:33:57.955838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:33:57.957437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:33:57.958146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:33:57.958445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:33:57.958733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:33:57.960975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:33:58.067999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:33:58.069412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0a82f6-e537-4b51-ae72-c6fe659d04f7 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:33:58.087731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 754/1487] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:33:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:34:00.328823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9e0dbd-fad8-47f4-b03c-de6170c1457f None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:34:00.329334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9e0dbd-fad8-47f4-b03c-de6170c1457f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:34:00.329334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9e0dbd-fad8-47f4-b03c-de6170c1457f None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:34:00.330658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 734/1488] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:34:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:34:00.336042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ec705-f4b7-40ff-99a8-03e1f350c6ee None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:34:00.336463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ec705-f4b7-40ff-99a8-03e1f350c6ee None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:34:00.336596 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ec705-f4b7-40ff-99a8-03e1f350c6ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:34:00.402706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5e7ec705-f4b7-40ff-99a8-03e1f350c6ee None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:34:00.403965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5e7ec705-f4b7-40ff-99a8-03e1f350c6ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:34:00.456478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:34:00.500393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 755/1489] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:34:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:34:00.787316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9e0dbd-fad8-47f4-b03c-de6170c1457f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:34:00.788855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1c9e0dbd-fad8-47f4-b03c-de6170c1457f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:34:00.825179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:34:00.890946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9e0dbd-fad8-47f4-b03c-de6170c1457f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:34:00.967555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:34:00.968158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:34:00.968436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:34:00.968706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:34:00.970304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:34:01.048123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:34:01.077649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:34:01.079104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab269200-63e9-486c-9365-2a0def027e45 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:34:01.096730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 735/1490] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:34:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:34:03.419747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450cf02-49d4-4284-aa5a-8a5fd9401c09 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:34:03.420325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450cf02-49d4-4284-aa5a-8a5fd9401c09 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:34:03.420325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450cf02-49d4-4284-aa5a-8a5fd9401c09 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:34:03.421625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 756/1491] 10.222.0.20 () {54 vars in 927 bytes} [Fri Sep 9 04:34:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:34:03.426896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edfa669-d462-4bc5-989f-bcf94b6c7c3c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:34:03.427317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edfa669-d462-4bc5-989f-bcf94b6c7c3c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:34:03.427510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edfa669-d462-4bc5-989f-bcf94b6c7c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:34:03.491794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5edfa669-d462-4bc5-989f-bcf94b6c7c3c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:34:03.493001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5edfa669-d462-4bc5-989f-bcf94b6c7c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:34:03.586571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 736/1492] 10.222.0.20 () {58 vars in 1024 bytes} [Fri Sep 9 04:34:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:34:03.883045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450cf02-49d4-4284-aa5a-8a5fd9401c09 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:34:03.884570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0450cf02-49d4-4284-aa5a-8a5fd9401c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:34:03.987705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450cf02-49d4-4284-aa5a-8a5fd9401c09 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:34:04.061854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:34:04.062822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:34:04.062822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:34:04.063059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:34:04.064581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:34:04.073151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:34:04.173689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:34:04.175141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835798f-2bfc-4472-b45c-a48b1d92faf3 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:34:04.193069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 757/1493] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:34:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:34:13.585028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=82 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:34:14.190738 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=50 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:23.505128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:23.505722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:23.505722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:23.580368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:35:23.581519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:23.678632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 737/1494] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:35:23.686898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:23.687423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:23.687544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:23.762563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:35:23.763807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:23.858481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 758/1495] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:35:23.862998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:23.864211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:23.875510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49a6493-b6a2-4c03-ba6f-0f877250b4a6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:23.935350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:23.969638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:23.970237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:23.970517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:23.970789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:23.971476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:35:23.971778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:23.972795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:23.972795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:35:23.972795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:35:23.973673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:23.984609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 738/1496] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:35:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:35:23.992862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:24.001960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd757b0-95f2-47d6-946b-45c2fb41db87 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:24.062181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:24.095839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:24.096434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.096713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.096982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.097617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:24.099365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:24.099365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:35:24.099365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:35:24.100576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:24.155400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 759/1497] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:35:23 2022] POST /identity/v3/projects => generated 379 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:35:24.159621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:24.161222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:24.194296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d926973-5ec5-41c2-b3f0-d6bedfe141ce admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:24.286584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:24.287215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.287492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.287758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.288398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:24.289410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:24.289410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:35:24.289410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:35:24.290596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:24.332347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 739/1498] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:35:24 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:35:24.336464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:24.338013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:24.399187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9971c24-b829-4934-9317-d97e8d5b29ad admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:24.482586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:24.494958 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:24.495582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.495861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.496133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.496770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:24.497766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:24.497766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:35:24.497766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:35:24.498646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:24.511342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 760/1499] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:35:24.515427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:24.516995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:24.528124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f2995-00d7-461f-a057-4d0a1bda66b9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:24.552626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:24.622916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:24.623546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.623823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.624094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.624735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:24.625753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:24.625753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:35:24.625753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:35:24.626634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8e8352-3083-469d-95c2-7eab6f64541d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:24.638725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 740/1500] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:35:24.642860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:24.644418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:24.681056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6474b3f1-d14a-45f8-a439-b277e5225335 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:24.775586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:24.776271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.776522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.776791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] PATH_INFO: `/v3/projects/f0ea9067e1734165b1516c6336052d9d/users/b68f93a4bd984eef91281de820728663/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.798912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0ea9067e1734165b1516c6336052d9d, user_id=b68f93a4bd984eef91281de820728663, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:24.800018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:24.800018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:35:24.800018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:35:24.801937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12532fec-c9bf-4a10-91f8-075f0669a4e4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:24.826648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 761/1501] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:35:24 2022] PUT /identity/v3/projects/f0ea9067e1734165b1516c6336052d9d/users/b68f93a4bd984eef91281de820728663/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:35:24.839272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200463c-e1d7-4660-8dd1-18272fc398ba None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.839565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200463c-e1d7-4660-8dd1-18272fc398ba None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.839848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200463c-e1d7-4660-8dd1-18272fc398ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.895592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-1200463c-e1d7-4660-8dd1-18272fc398ba None None] MFA Rules not processed for user `b68f93a4bd984eef91281de820728663`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:35:24.896811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1200463c-e1d7-4660-8dd1-18272fc398ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:24.978885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 741/1502] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:35:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:35:24.991850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1020bb8-453f-4b26-b61c-4bbc4bf57e78 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.992141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1020bb8-453f-4b26-b61c-4bbc4bf57e78 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.992445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1020bb8-453f-4b26-b61c-4bbc4bf57e78 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:24.994267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 762/1503] 10.222.0.20 () {54 vars in 964 bytes} [Fri Sep 9 04:35:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:35:24.998817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e4a993-1f18-4407-8efc-3ff60be31aa0 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:24.999139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e4a993-1f18-4407-8efc-3ff60be31aa0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:24.999416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e4a993-1f18-4407-8efc-3ff60be31aa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:25.030514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:25.075370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-98e4a993-1f18-4407-8efc-3ff60be31aa0 None None] MFA Rules not processed for user `a8add98306764181b5ea1e8a1190bbde`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:35:25.076571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-98e4a993-1f18-4407-8efc-3ff60be31aa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:25.149482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 742/1504] 10.222.0.20 () {58 vars in 1061 bytes} [Fri Sep 9 04:35:24 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:35:25.157146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1020bb8-453f-4b26-b61c-4bbc4bf57e78 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:25.158635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f1020bb8-453f-4b26-b61c-4bbc4bf57e78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:25.207903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:25.263033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1020bb8-453f-4b26-b61c-4bbc4bf57e78 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:25.335191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:25.335796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:25.336077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:25.336349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:25.337972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:25.437357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:25.438760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:25.442246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:25.457146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 763/1505] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:35:25 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:35:25.502740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7254c5-565c-43b9-9a58-1fe11fdc9391 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:25.503074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7254c5-565c-43b9-9a58-1fe11fdc9391 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:25.503380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7254c5-565c-43b9-9a58-1fe11fdc9391 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:25.504669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 743/1506] 10.222.0.20 () {52 vars in 830 bytes} [Fri Sep 9 04:35:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:35:25.508278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:25.509512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:25.607166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f45a36e-cef8-46f3-a5c8-219cb743a194 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:25.701100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:25.701719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:25.701993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:25.702261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] PATH_INFO: `/v3/users/b68f93a4bd984eef91281de820728663` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:25.715328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b68f93a4bd984eef91281de820728663)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:25.716334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:25.716334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:35:25.716334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:35:25.717531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:25.726998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:25.784455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b68f93a4bd984eef91281de820728663'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:25.794054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b68f93a4bd984eef91281de820728663'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:25.817792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b68f93a4bd984eef91281de820728663'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:25.818251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b68f93a4bd984eef91281de820728663'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:25.832020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b68f93a4bd984eef91281de820728663'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:25.838092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 764/1507] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:35:25 2022] DELETE /identity/v3/users/b68f93a4bd984eef91281de820728663 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:35:25.842491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:25.842989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:25.843115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:25.900062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:35:25.901274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:25.975178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:25.987943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 744/1508] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:35:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:35:25.998127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:25.999389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:26.099707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ada8b1-04f3-4678-b00f-1daa64cc6734 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:26.154832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:26.176591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:26.177211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:26.177492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:26.177768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:26.179400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:26.288533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:26.290465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eaee91-6943-4a85-b590-4a2f4fbda608 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:26.308841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 765/1509] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:35:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:35:26.647874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:35:26.649148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:35:26.661251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba22850-a9e4-4a76-b03a-0edee07e18ed None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:35:26.758308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:35:26.759449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:35:26.759753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:35:26.760026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] PATH_INFO: `/v3/projects/f0ea9067e1734165b1516c6336052d9d` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:35:26.769950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0ea9067e1734165b1516c6336052d9d)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:35:26.771064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:35:26.771064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:35:26.771064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:35:26.772285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:35:26.830545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0ea9067e1734165b1516c6336052d9d', 'request_id': 'req-bf29cbad-fb5b-4646-954f-48b52b1912ce'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:26.841316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0ea9067e1734165b1516c6336052d9d', 'request_id': 'req-bf29cbad-fb5b-4646-954f-48b52b1912ce'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:26.847439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0ea9067e1734165b1516c6336052d9d', 'request_id': 'req-bf29cbad-fb5b-4646-954f-48b52b1912ce'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:26.876359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] The token cache is being invalidate because project f0ea9067e1734165b1516c6336052d9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:35:26.876656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:26.877098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf29cbad-fb5b-4646-954f-48b52b1912ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:35:26.881599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 745/1510] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:35:26 2022] DELETE /identity/v3/projects/f0ea9067e1734165b1516c6336052d9d => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:35:36.306275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=62 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:35:36.874405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:36:55.422402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2225b994-e9b2-45bf-a18e-8b34becfe0b4 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:55.423039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2225b994-e9b2-45bf-a18e-8b34becfe0b4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:55.423039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2225b994-e9b2-45bf-a18e-8b34becfe0b4 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:55.424331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 766/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:36:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:36:55.429157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f482302a-6cbc-4415-acd8-c378704e9306 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:55.429593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f482302a-6cbc-4415-acd8-c378704e9306 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:55.429748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f482302a-6cbc-4415-acd8-c378704e9306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:55.515648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f482302a-6cbc-4415-acd8-c378704e9306 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:36:55.516789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f482302a-6cbc-4415-acd8-c378704e9306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:36:55.624379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 746/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:36:55.774235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2225b994-e9b2-45bf-a18e-8b34becfe0b4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:36:55.776409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2225b994-e9b2-45bf-a18e-8b34becfe0b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:36:55.893221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2225b994-e9b2-45bf-a18e-8b34becfe0b4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:36:55.898582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99b1d26-c5f2-4bad-8959-c4c7415377af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:36:55.899215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99b1d26-c5f2-4bad-8959-c4c7415377af service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:55.899524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99b1d26-c5f2-4bad-8959-c4c7415377af service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:55.899796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99b1d26-c5f2-4bad-8959-c4c7415377af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:55.907727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99b1d26-c5f2-4bad-8959-c4c7415377af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:36:55.909284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99b1d26-c5f2-4bad-8959-c4c7415377af service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:36:55.918462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 767/1513] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:36:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:36:56.588938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76b0bd7-78ce-4dfb-bc4f-1dda1816a7bb None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:56.589489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76b0bd7-78ce-4dfb-bc4f-1dda1816a7bb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:56.589489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76b0bd7-78ce-4dfb-bc4f-1dda1816a7bb None None] PATH_INFO: `/v3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:56.590765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 747/1514] 10.222.0.17 () {54 vars in 925 bytes} [Fri Sep 9 04:36:56 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:36:56.597868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be238ece-682c-4594-bb97-cfc1dacd09e5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:56.598326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be238ece-682c-4594-bb97-cfc1dacd09e5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:56.598517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be238ece-682c-4594-bb97-cfc1dacd09e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:56.659366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-be238ece-682c-4594-bb97-cfc1dacd09e5 None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:36:56.660587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-be238ece-682c-4594-bb97-cfc1dacd09e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:36:56.729000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:36:56.734502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 768/1515] 10.222.0.17 () {58 vars in 1015 bytes} [Fri Sep 9 04:36:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:36:56.995380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9cfdf4-fd56-4474-8ea5-5e4674801460 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:56.995663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9cfdf4-fd56-4474-8ea5-5e4674801460 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:56.995938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9cfdf4-fd56-4474-8ea5-5e4674801460 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:56.997280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 748/1516] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:36:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:36:57.003224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f9a5d-945c-4e71-a1d0-12e079779224 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:57.003509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f9a5d-945c-4e71-a1d0-12e079779224 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:57.003783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f9a5d-945c-4e71-a1d0-12e079779224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:57.023470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c90f9a5d-945c-4e71-a1d0-12e079779224 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:36:57.025263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c90f9a5d-945c-4e71-a1d0-12e079779224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:36:57.032959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 769/1517] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:36:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:36:57.184253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9cfdf4-fd56-4474-8ea5-5e4674801460 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:36:57.199288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9cfdf4-fd56-4474-8ea5-5e4674801460 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:36:57.204677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f85a88f-bcd7-496b-9f42-78b719153cc1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:36:57.205266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85a88f-bcd7-496b-9f42-78b719153cc1 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:36:57.205547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85a88f-bcd7-496b-9f42-78b719153cc1 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:36:57.205823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85a88f-bcd7-496b-9f42-78b719153cc1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:36:57.213728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f85a88f-bcd7-496b-9f42-78b719153cc1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:36:57.215340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f85a88f-bcd7-496b-9f42-78b719153cc1 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:36:57.224449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 749/1518] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:36:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:05.481456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db678a5-97dc-4f96-8d73-4651dd54c863 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:05.481945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db678a5-97dc-4f96-8d73-4651dd54c863 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:05.482056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db678a5-97dc-4f96-8d73-4651dd54c863 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:05.483446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 770/1519] 10.222.0.20 () {54 vars in 961 bytes} [Fri Sep 9 04:37:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:05.488357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453b44e9-456d-4097-9fcc-9bdc711ffc01 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:05.488781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453b44e9-456d-4097-9fcc-9bdc711ffc01 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:05.488895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453b44e9-456d-4097-9fcc-9bdc711ffc01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:05.508156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-453b44e9-456d-4097-9fcc-9bdc711ffc01 None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:05.509332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-453b44e9-456d-4097-9fcc-9bdc711ffc01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:05.517713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 750/1520] 10.222.0.20 () {58 vars in 1058 bytes} [Fri Sep 9 04:37:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:05.525404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db678a5-97dc-4f96-8d73-4651dd54c863 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:05.539084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db678a5-97dc-4f96-8d73-4651dd54c863 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:05.544862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f834d0a4-357f-481b-a8c2-7ba88549f6ec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:05.545443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834d0a4-357f-481b-a8c2-7ba88549f6ec service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:05.545747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834d0a4-357f-481b-a8c2-7ba88549f6ec service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:05.546020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834d0a4-357f-481b-a8c2-7ba88549f6ec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:05.554203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f834d0a4-357f-481b-a8c2-7ba88549f6ec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:05.555828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f834d0a4-357f-481b-a8c2-7ba88549f6ec service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:05.565439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 771/1521] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:08.725348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd491865-c5aa-4f39-be4b-6f0f969e23ed None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:08.725922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd491865-c5aa-4f39-be4b-6f0f969e23ed None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:08.725922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd491865-c5aa-4f39-be4b-6f0f969e23ed None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:08.727927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 751/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:37:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:08.733816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3b649d-8626-49a8-87c7-74f9a379a389 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:08.734243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3b649d-8626-49a8-87c7-74f9a379a389 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:08.734483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3b649d-8626-49a8-87c7-74f9a379a389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:08.755204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9d3b649d-8626-49a8-87c7-74f9a379a389 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:08.756402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9d3b649d-8626-49a8-87c7-74f9a379a389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:08.764671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 772/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:37:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:08.918008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd491865-c5aa-4f39-be4b-6f0f969e23ed None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:08.932600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd491865-c5aa-4f39-be4b-6f0f969e23ed None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:08.938233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff751884-7e6e-44bc-93ba-5ba1574e5ca0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:08.938819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff751884-7e6e-44bc-93ba-5ba1574e5ca0 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:08.939130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff751884-7e6e-44bc-93ba-5ba1574e5ca0 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:08.939402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff751884-7e6e-44bc-93ba-5ba1574e5ca0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:08.947757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff751884-7e6e-44bc-93ba-5ba1574e5ca0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:08.949346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff751884-7e6e-44bc-93ba-5ba1574e5ca0 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:08.958576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 752/1524] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:10.034480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db83f9f7-ea58-45e6-ab77-d001fee4fe99 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:10.035084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db83f9f7-ea58-45e6-ab77-d001fee4fe99 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:10.035084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db83f9f7-ea58-45e6-ab77-d001fee4fe99 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:10.036437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 773/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 9 04:37:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:10.043039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbd81b8-a672-49fb-a2a9-4317044af843 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:10.043484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbd81b8-a672-49fb-a2a9-4317044af843 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:10.043582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbd81b8-a672-49fb-a2a9-4317044af843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:10.063050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4fbd81b8-a672-49fb-a2a9-4317044af843 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:10.064239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4fbd81b8-a672-49fb-a2a9-4317044af843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:10.071967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 753/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 9 04:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:10.223746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db83f9f7-ea58-45e6-ab77-d001fee4fe99 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:10.238609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db83f9f7-ea58-45e6-ab77-d001fee4fe99 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:10.243996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cf2a63-d843-4843-96d0-b364cd030ca6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:10.244585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cf2a63-d843-4843-96d0-b364cd030ca6 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:10.244879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cf2a63-d843-4843-96d0-b364cd030ca6 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:10.245137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cf2a63-d843-4843-96d0-b364cd030ca6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:10.253413 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cf2a63-d843-4843-96d0-b364cd030ca6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:10.255066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cf2a63-d843-4843-96d0-b364cd030ca6 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:10.265049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 774/1527] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:14.208829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1368c55c-519f-49b0-9340-905a6b0a0cf7 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:14.209387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1368c55c-519f-49b0-9340-905a6b0a0cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:14.209387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1368c55c-519f-49b0-9340-905a6b0a0cf7 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:14.210727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 754/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:37:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:14.216510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85525fa-4dff-4ed5-a8db-313afc2a0d27 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:14.216929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85525fa-4dff-4ed5-a8db-313afc2a0d27 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:14.217033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85525fa-4dff-4ed5-a8db-313afc2a0d27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:14.236187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b85525fa-4dff-4ed5-a8db-313afc2a0d27 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:14.237376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b85525fa-4dff-4ed5-a8db-313afc2a0d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:14.245100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 775/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:14.533192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1368c55c-519f-49b0-9340-905a6b0a0cf7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:14.534903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1368c55c-519f-49b0-9340-905a6b0a0cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:14.569845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:37:14.613971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1368c55c-519f-49b0-9340-905a6b0a0cf7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:14.618144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bd19c4-890d-4f5d-ad5c-b921d7bdbcc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:14.618740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bd19c4-890d-4f5d-ad5c-b921d7bdbcc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:14.619048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bd19c4-890d-4f5d-ad5c-b921d7bdbcc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:14.619322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bd19c4-890d-4f5d-ad5c-b921d7bdbcc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:14.627989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bd19c4-890d-4f5d-ad5c-b921d7bdbcc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:14.629405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bd19c4-890d-4f5d-ad5c-b921d7bdbcc4 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:14.639181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 755/1530] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:16.779359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460b7ec-f734-40e7-b46d-8320fc14b4e9 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:16.779901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460b7ec-f734-40e7-b46d-8320fc14b4e9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:16.780007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460b7ec-f734-40e7-b46d-8320fc14b4e9 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:16.781346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 776/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:37:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:16.786634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617c531-858d-462d-a68e-99ac4f9ab60b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:16.787097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617c531-858d-462d-a68e-99ac4f9ab60b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:16.787209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617c531-858d-462d-a68e-99ac4f9ab60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:16.806966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3617c531-858d-462d-a68e-99ac4f9ab60b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:16.808174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3617c531-858d-462d-a68e-99ac4f9ab60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:16.816174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 756/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:17.104797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5460b7ec-f734-40e7-b46d-8320fc14b4e9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:17.115759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5460b7ec-f734-40e7-b46d-8320fc14b4e9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:17.119897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690cd3a9-b79f-4ab1-b1ac-289b3063d610 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:17.120485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690cd3a9-b79f-4ab1-b1ac-289b3063d610 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:17.120760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690cd3a9-b79f-4ab1-b1ac-289b3063d610 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:17.121028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690cd3a9-b79f-4ab1-b1ac-289b3063d610 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:17.129693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690cd3a9-b79f-4ab1-b1ac-289b3063d610 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:17.131130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690cd3a9-b79f-4ab1-b1ac-289b3063d610 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:17.140367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 777/1533] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:20.700662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33230b0-ac57-458c-9626-c6229121fd7d None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:20.701260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33230b0-ac57-458c-9626-c6229121fd7d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:20.701260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33230b0-ac57-458c-9626-c6229121fd7d None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:20.702546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 757/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:37:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:20.707891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c620db-435f-4e49-8d74-0859aa12bb38 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:20.708332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c620db-435f-4e49-8d74-0859aa12bb38 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:20.708477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c620db-435f-4e49-8d74-0859aa12bb38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:20.728193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d7c620db-435f-4e49-8d74-0859aa12bb38 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:20.729382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d7c620db-435f-4e49-8d74-0859aa12bb38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:20.737474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 778/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:21.040713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33230b0-ac57-458c-9626-c6229121fd7d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:21.042405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a33230b0-ac57-458c-9626-c6229121fd7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:21.125103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33230b0-ac57-458c-9626-c6229121fd7d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:21.130024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ac027e-ae3c-409f-9d35-274d0687352f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:21.131115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ac027e-ae3c-409f-9d35-274d0687352f service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:21.131427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ac027e-ae3c-409f-9d35-274d0687352f service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:21.131709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ac027e-ae3c-409f-9d35-274d0687352f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:21.140892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ac027e-ae3c-409f-9d35-274d0687352f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:21.142359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ac027e-ae3c-409f-9d35-274d0687352f service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:21.152212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 758/1536] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:23.299555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bbe3a7-4fa2-4e80-9325-91aceec7d56c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:23.300101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bbe3a7-4fa2-4e80-9325-91aceec7d56c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:23.300101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bbe3a7-4fa2-4e80-9325-91aceec7d56c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:23.301436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 779/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 9 04:37:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:37:23.306559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb74386e-f4d8-4e1e-93ec-532709447f88 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:23.307031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb74386e-f4d8-4e1e-93ec-532709447f88 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:23.307179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb74386e-f4d8-4e1e-93ec-532709447f88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:23.327159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-eb74386e-f4d8-4e1e-93ec-532709447f88 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:37:23.328351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-eb74386e-f4d8-4e1e-93ec-532709447f88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:23.337015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 759/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 9 04:37:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:37:23.643895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bbe3a7-4fa2-4e80-9325-91aceec7d56c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:37:23.645547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-78bbe3a7-4fa2-4e80-9325-91aceec7d56c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:37:23.658151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bbe3a7-4fa2-4e80-9325-91aceec7d56c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:37:23.661627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425bdd19-0e40-41cc-9832-b48510536320 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:37:23.662209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425bdd19-0e40-41cc-9832-b48510536320 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:37:23.662485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425bdd19-0e40-41cc-9832-b48510536320 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:37:23.662754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425bdd19-0e40-41cc-9832-b48510536320 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:37:23.671469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425bdd19-0e40-41cc-9832-b48510536320 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:37:23.672866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425bdd19-0e40-41cc-9832-b48510536320 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:37:23.683500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 780/1539] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:37:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:37:33.322352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=63 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:37:33.677919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=37 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:48.985033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:48.986283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:48.986795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.013993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.015279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.024594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 760/1540] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.033856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ffb90-9899-4f8c-93c5-76b75f84ed61 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.034495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ffb90-9899-4f8c-93c5-76b75f84ed61 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.034495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ffb90-9899-4f8c-93c5-76b75f84ed61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.064300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-230ffb90-9899-4f8c-93c5-76b75f84ed61 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.065544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-230ffb90-9899-4f8c-93c5-76b75f84ed61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.075559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 781/1541] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.081644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.092402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fe3060-3a90-4e6d-9cf8-374eaa7c3116 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.096318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548b48ba-0225-48b9-934d-d6bb6ccf1f86 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.096610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548b48ba-0225-48b9-934d-d6bb6ccf1f86 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.096894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548b48ba-0225-48b9-934d-d6bb6ccf1f86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.098980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.099655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.099932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.100205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.100867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:49.101173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.104571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.119256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-548b48ba-0225-48b9-934d-d6bb6ccf1f86 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.120622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-548b48ba-0225-48b9-934d-d6bb6ccf1f86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.125511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 761/1542] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:49.134662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.137949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 782/1543] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:38:49.147139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.147587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.147873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.150608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9d6bb4-ad78-4cae-b548-7988eb1fe167 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.157068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.157697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.158123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.158560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.159341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.160489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:38:49.160489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:38:49.160489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:38:49.161723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.182722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.183980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.200410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 783/1544] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:38:49.204451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.220461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ea70f0-ad00-4e65-8085-bc2fa108636f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.229690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.230474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.230794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.231277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.235340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:49.235572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.236898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:38:49.236898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:38:49.236898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:38:49.237910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2334c00-58d6-4978-b291-a075460ddd85 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.249369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 762/1545] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/projects => generated 393 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:49.252878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.256002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 784/1546] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:38:49.257673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.258075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.258357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.290102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.293851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.379245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3010c36-c68c-45ae-9d51-2205654791dc admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.398045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f807ba36-6f6d-464f-a369-0fa9084d6457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.398045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f807ba36-6f6d-464f-a369-0fa9084d6457 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.398380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f807ba36-6f6d-464f-a369-0fa9084d6457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.398642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f807ba36-6f6d-464f-a369-0fa9084d6457 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.400854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 785/1547] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.401502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.402180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f807ba36-6f6d-464f-a369-0fa9084d6457 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.404406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f807ba36-6f6d-464f-a369-0fa9084d6457 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.423063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e82c41-2ff5-4352-add5-63d07913cf81 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.435397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc740ec-ead0-469e-ba0d-d20679e2861e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.436450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc740ec-ead0-469e-ba0d-d20679e2861e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.436948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc740ec-ead0-469e-ba0d-d20679e2861e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.437591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc740ec-ead0-469e-ba0d-d20679e2861e admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.438758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc740ec-ead0-469e-ba0d-d20679e2861e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.441817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc740ec-ead0-469e-ba0d-d20679e2861e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.461552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 763/1548] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/users => generated 297 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:49.463268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3addb8db-add8-4805-b97e-7f04036cc4db None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.463655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3addb8db-add8-4805-b97e-7f04036cc4db None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.463968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3addb8db-add8-4805-b97e-7f04036cc4db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.496178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3addb8db-add8-4805-b97e-7f04036cc4db None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.497205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 786/1549] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/projects => generated 389 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:49.498130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3addb8db-add8-4805-b97e-7f04036cc4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.498668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.501577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.501577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.529054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.530277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.576527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 764/1550] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.578314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.578806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.579326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.608929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 787/1551] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.610045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.610857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.612121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.622176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 765/1552] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.626010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.627851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c871f5-d92d-46fe-b60f-538acd9d0d86 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.633570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a6c116-f19a-40e1-9583-52bac16c7560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.634190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a6c116-f19a-40e1-9583-52bac16c7560 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.634472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a6c116-f19a-40e1-9583-52bac16c7560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.634749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a6c116-f19a-40e1-9583-52bac16c7560 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.635432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a6c116-f19a-40e1-9583-52bac16c7560 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.635554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9298a4d4-0df0-4ffc-b3cc-bc40bdf03f7a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.637238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a6c116-f19a-40e1-9583-52bac16c7560 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.640416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756f537-41f8-4a70-ad83-aaf0ed8c178e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.641020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756f537-41f8-4a70-ad83-aaf0ed8c178e admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.641296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756f537-41f8-4a70-ad83-aaf0ed8c178e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.641567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756f537-41f8-4a70-ad83-aaf0ed8c178e admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.642201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0756f537-41f8-4a70-ad83-aaf0ed8c178e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.644133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0756f537-41f8-4a70-ad83-aaf0ed8c178e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.649434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 788/1553] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:49.649434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d85d88-0e20-474e-b101-63fe582baf50 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.649434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d85d88-0e20-474e-b101-63fe582baf50 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.649729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d85d88-0e20-474e-b101-63fe582baf50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.666378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 766/1554] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/users => generated 295 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:49.667928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9801b0-21b8-473e-9380-b8a543af54fc None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.668215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9801b0-21b8-473e-9380-b8a543af54fc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.668492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9801b0-21b8-473e-9380-b8a543af54fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.670401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-26d85d88-0e20-474e-b101-63fe582baf50 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.671633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-26d85d88-0e20-474e-b101-63fe582baf50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.680026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 789/1555] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.681194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.681476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.681746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.689716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-dc9801b0-21b8-473e-9380-b8a543af54fc None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.691281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dc9801b0-21b8-473e-9380-b8a543af54fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.699991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 767/1556] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.701244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431a988c-ee48-44f2-8e54-0aa18ac4a26a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.701526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431a988c-ee48-44f2-8e54-0aa18ac4a26a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.701794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431a988c-ee48-44f2-8e54-0aa18ac4a26a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.702637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.703860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.711726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 790/1557] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.712496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.721946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439f4eee-b64c-4537-9f99-05865d5a909f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.723203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-431a988c-ee48-44f2-8e54-0aa18ac4a26a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.724423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-431a988c-ee48-44f2-8e54-0aa18ac4a26a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.726598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.727243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.727522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.727798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.728441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:49.728743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.730358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.732741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 768/1558] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.733915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.734205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.734483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.742507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:49.742661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 791/1559] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:49.743579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.760541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cc5cd4-1cb7-4bf1-8a13-39bc86abe1a8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.762308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.763552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.765042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16600820-1d73-4c46-ac87-61e5b1a9f6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.765635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16600820-1d73-4c46-ac87-61e5b1a9f6d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.765911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16600820-1d73-4c46-ac87-61e5b1a9f6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.766187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16600820-1d73-4c46-ac87-61e5b1a9f6d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.766827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16600820-1d73-4c46-ac87-61e5b1a9f6d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.768448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16600820-1d73-4c46-ac87-61e5b1a9f6d9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.770640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:49.772509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 769/1560] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.773554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.783493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 792/1561] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:49.784775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4919d3-071f-4114-995e-f1087307dee8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.785295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4919d3-071f-4114-995e-f1087307dee8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.786048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4919d3-071f-4114-995e-f1087307dee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.786294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d23569-12b6-45f6-90a1-6b6f3f848bd4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.792170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.792773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.793047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.793314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.793956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.795577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.805724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 770/1562] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:49.806351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0b4919d3-071f-4114-995e-f1087307dee8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.806573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.807598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0b4919d3-071f-4114-995e-f1087307dee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.816065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 793/1563] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.816250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16f226c-21b2-4c12-93e0-fb565f4cc12e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.817241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.817528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.817824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.821431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.822028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.822303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.822571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.823317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:49.823619 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.825189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.837587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 771/1564] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:49.837902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.838471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.839313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.847286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 794/1565] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.847829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c44a69f-824d-41e0-8794-c79bd635dbc4 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.848147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.853325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.854086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.854371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.855202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.855846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:49.856310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.857899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb4ddce-8f7c-4e02-8a7f-2b0ef0157b33 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.858194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1abc15-c384-49bc-ae3a-7a0a40131196 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.862749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.863352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.863627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.863894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.864511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.866420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.869608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 772/1566] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:49.872933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.873216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.873487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.893237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:49.894441 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:49.897918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 795/1567] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:49.898721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.954880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 773/1568] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:49.955754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.961610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e11cf-afd7-4fa3-b200-30d1e06207e0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.966259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.966909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.967216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.967494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] PATH_INFO: `/v3/projects/1d203c97242044fb8bf29bdc461e2efd/users/d3c29c23d7524adf9c5a3271224ac8d3/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.967970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663e6db2-033e-43d5-bd09-b94987bbf169 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:49.973129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:49.973719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:49.974002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:49.974277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:49.974971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.976612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.981981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d203c97242044fb8bf29bdc461e2efd, user_id=d3c29c23d7524adf9c5a3271224ac8d3, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:49.984715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:49.986586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 774/1569] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:49.987257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:49.997656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34bc98-b18d-46d7-9990-3984a154f8f9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.000480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 796/1570] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:49 2022] PUT /identity/v3/projects/1d203c97242044fb8bf29bdc461e2efd/users/d3c29c23d7524adf9c5a3271224ac8d3/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:50.004404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.053370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.053897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.054174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.054445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.055113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:50.055415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.057000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.064663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8499dcd3-f416-4fc6-bc82-217478ecb058 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.069459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 775/1571] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:50.077000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.077119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.077221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.077221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.077221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.077221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.077221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.081400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc03ec-4c49-4206-a0ac-b86c94637b59 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.086571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.087186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.087465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.087737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.088376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:50.088680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.090251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.105644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 797/1572] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/projects => generated 375 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.105858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 776/1573] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:50.105984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.106095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.115041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c38065-ddf6-4b57-b798-fb77d8a7e94c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.140931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:50.143022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:50.172154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.172759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.173035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.173172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da20e0-5d0c-4ee1-994d-ff13c6176785 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.173290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.173959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.176046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.178089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.178688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.178998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.179274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.179895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.181771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.197381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 777/1574] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.198331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.207874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ed6ed6-e1f3-41aa-a6ab-2d69e8cb3921 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.212965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.213152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 798/1575] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/projects => generated 371 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.213555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.213863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.214102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.214206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.214873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:50.215204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.216803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c15b5f-f6ca-4e79-8fcb-f5e885c79628 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.231194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 778/1576] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:50.236587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.236945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.237632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.256878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:50.258104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:50.282876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a01e581-9bda-41d0-96df-ffb822d0db15 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.287536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.288191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.288548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.288747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] PATH_INFO: `/v3/projects/5176b458a4a8481eaa37d8693b51f15e/users/befee01f638c4e7681b30c64acad6a6a/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.302953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5176b458a4a8481eaa37d8693b51f15e, user_id=befee01f638c4e7681b30c64acad6a6a, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.305675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.319123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 799/1577] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:50 2022] PUT /identity/v3/projects/5176b458a4a8481eaa37d8693b51f15e/users/befee01f638c4e7681b30c64acad6a6a/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:50.319954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.324228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 779/1578] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:50.324228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.332215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faba233-9af8-4cca-b28a-b03b777e2b2b None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.387784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc09f6-ad5b-4976-9b6c-2266f719ee04 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.388609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.389231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.389511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.389954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.390624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.392645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.392754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.393329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.393608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.393882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.394511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.396451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.415990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 780/1579] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.416857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.426794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1204fb-72f8-41a0-b464-11066b5c73af admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.427470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 800/1580] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/projects => generated 375 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.428895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.460003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:50.462968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:50.491124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.491741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.492024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.492310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.492947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.494862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.497499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e626903-9ab0-41a6-acb0-f911108c2813 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.502206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.502809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.503118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.503390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.504030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.505593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.516965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 801/1581] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:50.517767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.525193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 781/1582] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/projects => generated 387 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.525193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.527446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f819d15-57f4-468b-b587-a302b4d7d404 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.583532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.584188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.584524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.584797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.585419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.586423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:38:50.586423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:38:50.586423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:38:50.586789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0033fefa-9064-43ed-b48b-2c95c3426ba1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.587657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c790f9e5-86a8-4353-b322-34aeeac99ae2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.591458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.592066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.592351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.592630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.593265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.595361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.609372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 802/1583] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/users => generated 286 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.613127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.613587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.613908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.628789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 782/1584] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/projects => generated 403 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.628789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.636162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:50.637352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:50.690593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9824cda2-27e0-48cc-a488-03d330d28e5c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.695016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.695739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.696025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.696455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.697999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 803/1585] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:50.698836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.706172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.708039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d3f7f6-03de-4fd7-a2a9-dabba91f3e89 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.708179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.713153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.713880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.714169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.714446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.715122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.716700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.718839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 783/1586] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:50.723815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.729933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 804/1587] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:50.729933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.735893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c75ce28-c12d-428c-8679-09ae9eb5d228 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.741959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0562b4c-85d0-4ab0-a61f-6e4379eaf073 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.742289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.742981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.743273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.743655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.744496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.747205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.747296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.747838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.748121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.748398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.749055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.750626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.761941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 805/1588] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:50.763026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.775101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa57494-1682-4d74-bab7-08711330aa14 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.779861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 784/1589] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/users => generated 288 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.779861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.783803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.784448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.784733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.785016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.785660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.787813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.790878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e9803d-841c-4a2d-8bee-b82d4eef07c1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.796906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.798213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.799226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.799585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.800648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.802313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.811772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 806/1590] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/users => generated 294 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.812818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.814946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 785/1591] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:50.815867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.824376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b14b0ba-4a1c-4f66-be8c-ec945c1b969f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.829038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.829790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.830093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.830352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.830493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3fdbeb-170e-4aa9-a9e1-fd64d1091b70 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.831021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.832950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.835142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.835876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.836257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.836607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.844479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:50.849668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.851495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.862414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 807/1592] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:50 2022] POST /identity/v3/users => generated 302 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:50.862977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.864743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 786/1593] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:50.866291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.873795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29a58eb-3ea2-41c9-b7fb-66d637f38882 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.877411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15dd4a2-22da-4d6b-8d33-9a20478f6107 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.879010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.879763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.880054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.880350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.881720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.883357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.884661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.885396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.885780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.886194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] PATH_INFO: `/v3/projects/bb415f49dfd64fdc88c46543878c6e74/users/7169599dbb3e47f098cacbfe1ae5bfcc/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.889009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:50.897746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 808/1594] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:50.898790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.906835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb415f49dfd64fdc88c46543878c6e74, user_id=7169599dbb3e47f098cacbfe1ae5bfcc, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.908080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:38:50.908080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:38:50.908080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:38:50.913377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.913484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbd039d-5551-4eea-95cd-2276cac42cf1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:50.918199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:50.919234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:50.919787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:50.920170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:50.921452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:50.924086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:50.931472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 787/1595] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:50 2022] PUT /identity/v3/projects/bb415f49dfd64fdc88c46543878c6e74/users/7169599dbb3e47f098cacbfe1ae5bfcc/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:50.932641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.936209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 809/1596] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:50.937374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:50.953585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad83a6-754e-425c-b2a9-eae874aae18f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.020489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.021191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.022149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.022497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.023261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.024933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.029031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2697d5-d027-4870-b4f8-f60e2510c693 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.036632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.037487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.037959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 810/1597] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:51.038305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.038775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.039073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.039983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.042017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.049411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65df15a4-9397-44a6-a30e-bbd94829cf11 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.060312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 788/1598] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:50 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:51.060524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.060723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.060897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.060897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.060897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.060897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.063104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.073846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9aed47-4c90-4965-ac41-48aa92b113d9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.079919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254e2902-7cec-481a-8f21-bb805e2bf8c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.080727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254e2902-7cec-481a-8f21-bb805e2bf8c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.081908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254e2902-7cec-481a-8f21-bb805e2bf8c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.081908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254e2902-7cec-481a-8f21-bb805e2bf8c0 admin admin] PATH_INFO: `/v3/projects/8cc9340f30fa4e299fd096d27b9c7dbd/users/161e111ce6384c0caadaa06d4bcffee2/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.085539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 811/1599] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:51 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:51.087564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.104583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254e2902-7cec-481a-8f21-bb805e2bf8c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cc9340f30fa4e299fd096d27b9c7dbd, user_id=161e111ce6384c0caadaa06d4bcffee2, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.105341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a073bdd-4d1d-4b68-9e45-0f1b81547e3c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.108009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254e2902-7cec-481a-8f21-bb805e2bf8c0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.114469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.116423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.116959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.117525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.119054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.122169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.136864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 789/1600] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:51 2022] PUT /identity/v3/projects/8cc9340f30fa4e299fd096d27b9c7dbd/users/161e111ce6384c0caadaa06d4bcffee2/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:51.139205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cb0dca-c5c4-4a85-ade7-171feb4186ee None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.140571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cb0dca-c5c4-4a85-ade7-171feb4186ee None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.141022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cb0dca-c5c4-4a85-ade7-171feb4186ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.150562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 812/1601] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:51 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:51.152180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.175248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0beae-9fb9-448f-b434-2a8fe922f858 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.188039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-75cb0dca-c5c4-4a85-ade7-171feb4186ee None None] MFA Rules not processed for user `7169599dbb3e47f098cacbfe1ae5bfcc`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:51.190861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-75cb0dca-c5c4-4a85-ade7-171feb4186ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:51.316439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.319118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.319630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.320103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.321317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.326677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.351717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 813/1602] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:51 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:51.351717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.369732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1271fa4-534a-42cf-b3d8-47614faaf809 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.375997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.376816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.377203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.378016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] PATH_INFO: `/v3/projects/69966549f30d4d25ade9bdaa804da7ae/users/2319df8bb7c94393b41765cbe4072f4d/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.400434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=69966549f30d4d25ade9bdaa804da7ae, user_id=2319df8bb7c94393b41765cbe4072f4d, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.405600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.429313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 814/1603] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:51 2022] PUT /identity/v3/projects/69966549f30d4d25ade9bdaa804da7ae/users/2319df8bb7c94393b41765cbe4072f4d/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:51.430475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.436463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 790/1604] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:51.438008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.438735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.440385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.476354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:51.502185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:51.503685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:51.567421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac301a4a-4e70-44c7-a897-75206677cd2e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.575843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.575843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.576196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.576858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.577664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.579408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.593404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 815/1605] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:51 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:51.593708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.601908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:51.617024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b0b94-7070-48e8-bcfa-2a39ebba8289 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.626962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2018d263-d576-465c-9c33-69627901cc89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.628021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2018d263-d576-465c-9c33-69627901cc89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.628308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2018d263-d576-465c-9c33-69627901cc89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.628709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2018d263-d576-465c-9c33-69627901cc89 admin admin] PATH_INFO: `/v3/projects/7ad09c9b062641c19d3811d5c96a24f8/users/bf68e214fd4740be96f2d6c4325f6ee9/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.657609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2018d263-d576-465c-9c33-69627901cc89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ad09c9b062641c19d3811d5c96a24f8, user_id=bf68e214fd4740be96f2d6c4325f6ee9, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.659786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 791/1606] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:51.660800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2018d263-d576-465c-9c33-69627901cc89 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.660927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.675788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52f02ab-a356-4901-ada0-35f0b4e507f8 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.685992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 816/1607] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:51 2022] PUT /identity/v3/projects/7ad09c9b062641c19d3811d5c96a24f8/users/bf68e214fd4740be96f2d6c4325f6ee9/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:51.688890 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.689447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.689955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.730671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] MFA Rules not processed for user `2319df8bb7c94393b41765cbe4072f4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:51.732325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:51.761768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab034e4d-0a7b-4ebf-a5fb-a79bacda1cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.762436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab034e4d-0a7b-4ebf-a5fb-a79bacda1cd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.762879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab034e4d-0a7b-4ebf-a5fb-a79bacda1cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.763216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab034e4d-0a7b-4ebf-a5fb-a79bacda1cd7 admin admin] PATH_INFO: `/v3/projects/574802c84aa7424b9f4a819d8107a7a9/users/b2c41a8904124aa6b31ae272a684b060/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.786720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab034e4d-0a7b-4ebf-a5fb-a79bacda1cd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=574802c84aa7424b9f4a819d8107a7a9, user_id=b2c41a8904124aa6b31ae272a684b060, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.792372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab034e4d-0a7b-4ebf-a5fb-a79bacda1cd7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.810262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 792/1608] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:51 2022] PUT /identity/v3/projects/574802c84aa7424b9f4a819d8107a7a9/users/b2c41a8904124aa6b31ae272a684b060/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:51.820114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd5229e-99ef-43a5-9a60-b0feb347d05c None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.820114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd5229e-99ef-43a5-9a60-b0feb347d05c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.820114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd5229e-99ef-43a5-9a60-b0feb347d05c None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.820114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 793/1609] 10.222.0.20 () {54 vars in 962 bytes} [Fri Sep 9 04:38:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:38:51.820114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd5229e-99ef-43a5-9a60-b0feb347d05c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.843694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 817/1610] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:51.844643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:51.907591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd5229e-99ef-43a5-9a60-b0feb347d05c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.920238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0830eaa-ab6c-402c-9879-bb78f0b8f0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:51.921500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0830eaa-ab6c-402c-9879-bb78f0b8f0be admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.921891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0830eaa-ab6c-402c-9879-bb78f0b8f0be admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.922276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0830eaa-ab6c-402c-9879-bb78f0b8f0be admin admin] PATH_INFO: `/v3/projects/6dd9e6c08659499c864d7f799f07fa78/users/225e6103cfc144edbc304779e2878f82/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.939902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d10d91-19f4-4f86-918a-55fe011a79a8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:51.952708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0830eaa-ab6c-402c-9879-bb78f0b8f0be admin admin] RBAC: Authorizing `identity:create_grant(project_id=6dd9e6c08659499c864d7f799f07fa78, user_id=225e6103cfc144edbc304779e2878f82, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:51.955892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0830eaa-ab6c-402c-9879-bb78f0b8f0be admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:51.979786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 794/1611] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:51 2022] PUT /identity/v3/projects/6dd9e6c08659499c864d7f799f07fa78/users/225e6103cfc144edbc304779e2878f82/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:51.981710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22c2ad-7b0c-4163-9e32-4fb1b9687287 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:51.982275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22c2ad-7b0c-4163-9e32-4fb1b9687287 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:51.982425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22c2ad-7b0c-4163-9e32-4fb1b9687287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:51.984095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:52.011882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a42a313-ed60-41ff-b3ba-187959a4f723 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.013013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a42a313-ed60-41ff-b3ba-187959a4f723 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.013505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a42a313-ed60-41ff-b3ba-187959a4f723 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.014165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a42a313-ed60-41ff-b3ba-187959a4f723 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.015814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5f22c2ad-7b0c-4163-9e32-4fb1b9687287 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:52.017724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f22c2ad-7b0c-4163-9e32-4fb1b9687287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:52.027178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a42a313-ed60-41ff-b3ba-187959a4f723 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.029214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a42a313-ed60-41ff-b3ba-187959a4f723 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.043957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 818/1612] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:52.049790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.050892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.051389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.095510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:52.103918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:52.105577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:52.123316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 795/1613] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:52.126721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ddd7f1-35f2-4030-a7e5-10a9297b4d1c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.126721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ddd7f1-35f2-4030-a7e5-10a9297b4d1c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.126721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ddd7f1-35f2-4030-a7e5-10a9297b4d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.164855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-27ddd7f1-35f2-4030-a7e5-10a9297b4d1c None None] MFA Rules not processed for user `b2c41a8904124aa6b31ae272a684b060`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:52.167423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-27ddd7f1-35f2-4030-a7e5-10a9297b4d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:52.211710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 819/1614] 10.222.0.20 () {58 vars in 1059 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:52.212602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.224955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d85dbf5-da4b-46c3-8f9a-2d48cad7d657 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.241762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.242398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.242726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.243036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.243785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.245939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.264276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 796/1615] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:52.265866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.266262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.266639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.292723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:52.294762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:52.296985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 820/1616] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/projects => generated 373 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:52.298980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.301428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:52.387943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f4d4b-adc1-4f47-bf6a-613b87ce5d31 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.392695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.393363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.393703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.394042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.395455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 797/1617] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:52.400059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.421855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.423902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.447635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 821/1618] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:52.448555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.459740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c28117-0e83-45f8-aa67-82c7c08d4b7d service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.478518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.479474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.479760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.480035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.480679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.482605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.542860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 822/1619] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/projects => generated 385 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:52.544682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.554329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247d1276-dc56-4fa5-8d5b-db197d7ca78c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.581786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:52.592045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:52.659936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcea156-e66d-4d67-b271-036a809afad9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.667631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.668272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.668597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.669189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.670058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.672807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.694904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.697234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.697939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.700019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.709858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 823/1620] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/users => generated 293 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:52.711176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.849173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9077c1a-a82a-4592-a44f-a2a4900c4e16 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.851150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.854374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.858851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.859508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.859795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.860068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.889427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 798/1621] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:52.893283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.900196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.908532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.939211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5cd059-ef4a-4127-b59b-49de34eaffff service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.939978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 824/1622] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:52.940986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:52.960203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.960869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.961151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.961419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.962063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.963571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033627de-bfef-42a5-90ad-c647a6e7b356 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:52.969564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:52.983909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:52.984600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:52.984940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:52.985434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:52.987637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:52.990825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.008720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 825/1623] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:52 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:53.010508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.020319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 799/1624] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:52 2022] POST /identity/v3/users => generated 293 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:53.024314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.026821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b347967-1f48-4913-a16d-70f6ea1e79a7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.040256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.042779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.044083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.044611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f35129-403e-4a17-a008-8105c662904f admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.045863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] PATH_INFO: `/v3/projects/6078f9184764475796b3c0e9bb413d45/users/2fa5026b16ed4d6a89c39613c345ec40/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.056786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.058983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.059280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.060748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.062621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.068128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.096061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 800/1625] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:53 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:53.097140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.105587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6078f9184764475796b3c0e9bb413d45, user_id=2fa5026b16ed4d6a89c39613c345ec40, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.117844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.120235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae88b1fa-4ebf-4165-9997-e19935b5824a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.140064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 826/1626] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:53 2022] PUT /identity/v3/projects/6078f9184764475796b3c0e9bb413d45/users/2fa5026b16ed4d6a89c39613c345ec40/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:53.141693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.142456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.142742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.143053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.144038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.145195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.145699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.186754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 801/1627] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:53 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:53.191074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.271830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420f2b8-c689-46fc-b2f0-1225f9adae16 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.339962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030f68c0-3d01-44f4-ab4e-d8de727b208e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.352995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.358709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.359127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.359422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.360366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.361567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c794a10-c52e-4951-850a-3c618d16ae98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.362252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c794a10-c52e-4951-850a-3c618d16ae98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.362363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.362529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c794a10-c52e-4951-850a-3c618d16ae98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.362805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c794a10-c52e-4951-850a-3c618d16ae98 admin admin] PATH_INFO: `/v3/projects/5e9e91fd716e45938154a7c743d003a5/users/a71d2db2a2294af084c1fcca30624801/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.397501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 827/1628] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:53 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:53.400097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.418594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c794a10-c52e-4951-850a-3c618d16ae98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e9e91fd716e45938154a7c743d003a5, user_id=a71d2db2a2294af084c1fcca30624801, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.421991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c794a10-c52e-4951-850a-3c618d16ae98 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.424309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba87e4-51e8-49f7-bbf8-319375007044 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.439960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:53.447338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1975e428-6a1d-4fcb-bbd5-8be2fc0bc730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.447939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975e428-6a1d-4fcb-bbd5-8be2fc0bc730 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.448225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975e428-6a1d-4fcb-bbd5-8be2fc0bc730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.448502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975e428-6a1d-4fcb-bbd5-8be2fc0bc730 admin admin] PATH_INFO: `/v3/projects/6078f9184764475796b3c0e9bb413d45/users/2fa5026b16ed4d6a89c39613c345ec40/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.454829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1975e428-6a1d-4fcb-bbd5-8be2fc0bc730 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6078f9184764475796b3c0e9bb413d45, user_id=2fa5026b16ed4d6a89c39613c345ec40, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.455836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 802/1629] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:53 2022] PUT /identity/v3/projects/5e9e91fd716e45938154a7c743d003a5/users/a71d2db2a2294af084c1fcca30624801/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:38:53.457416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.457706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.457982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.463384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1975e428-6a1d-4fcb-bbd5-8be2fc0bc730 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.470340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:53.489974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 828/1630] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:53 2022] PUT /identity/v3/projects/6078f9184764475796b3c0e9bb413d45/users/2fa5026b16ed4d6a89c39613c345ec40/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:38:53.500677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.501538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.501826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.531266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] MFA Rules not processed for user `a71d2db2a2294af084c1fcca30624801`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:53.532506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:53.570082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] MFA Rules not processed for user `2fa5026b16ed4d6a89c39613c345ec40`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:53.571466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:53.688175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 803/1631] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:53.689850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.778611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 829/1632] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:53 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:53.792943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.816724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801ce0a1-fd34-4fa3-8e62-d00a440091da None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.831480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d13157f-467e-4af2-8a05-fd7d4136b597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:53.832809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d13157f-467e-4af2-8a05-fd7d4136b597 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.833191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d13157f-467e-4af2-8a05-fd7d4136b597 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.833579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d13157f-467e-4af2-8a05-fd7d4136b597 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.834988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d13157f-467e-4af2-8a05-fd7d4136b597 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:53.839732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d13157f-467e-4af2-8a05-fd7d4136b597 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:53.879774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 804/1633] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:53 2022] POST /identity/v3/projects => generated 387 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:53.881913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7529fb67-0ce8-4eaf-9377-c5e84f9eaeb1 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:53.882202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7529fb67-0ce8-4eaf-9377-c5e84f9eaeb1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:53.882535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7529fb67-0ce8-4eaf-9377-c5e84f9eaeb1 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:53.889751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 805/1634] 10.222.0.20 () {54 vars in 962 bytes} [Fri Sep 9 04:38:53 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:38:53.890849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7529fb67-0ce8-4eaf-9377-c5e84f9eaeb1 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:53.894075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee9e5bc-a6a3-4559-bef6-8e2ca78608ca None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:53.992609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7529fb67-0ce8-4eaf-9377-c5e84f9eaeb1 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.004341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b202e-d3c9-41ad-b5ef-26da8e26dcf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.005315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b202e-d3c9-41ad-b5ef-26da8e26dcf9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.006456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b202e-d3c9-41ad-b5ef-26da8e26dcf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.006877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b202e-d3c9-41ad-b5ef-26da8e26dcf9 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.007827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b202e-d3c9-41ad-b5ef-26da8e26dcf9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.010987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b202e-d3c9-41ad-b5ef-26da8e26dcf9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.047463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 806/1635] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:53 2022] POST /identity/v3/users => generated 300 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:54.049500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf57d6d8-7136-4f1f-95ad-d21db052872d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.050361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf57d6d8-7136-4f1f-95ad-d21db052872d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.051210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf57d6d8-7136-4f1f-95ad-d21db052872d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.063005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:54.069862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.071844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.072660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.073275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.094064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bf57d6d8-7136-4f1f-95ad-d21db052872d None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:54.095405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bf57d6d8-7136-4f1f-95ad-d21db052872d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:54.191261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:54.223595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.227573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.272419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 830/1636] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:53 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:38:54.276674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:54.290989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 807/1637] 10.222.0.20 () {58 vars in 1059 bytes} [Fri Sep 9 04:38:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:54.299469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc47b5b-f10f-49fa-bd32-c8bda796096d None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.299776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc47b5b-f10f-49fa-bd32-c8bda796096d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.300053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc47b5b-f10f-49fa-bd32-c8bda796096d None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.301935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 808/1638] 10.222.0.20 () {54 vars in 912 bytes} [Fri Sep 9 04:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:38:54.310711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.311026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.311305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.359887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c46580d-a8de-4e02-bd20-2dfd18400c21 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.366830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:54.372297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:54.374690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.385821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.386109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.387998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.388691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.396903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.418368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 809/1639] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:38:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:54.438695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:54.449223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 831/1640] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:54 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:54.450206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:54.465002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf169854-d2f2-4254-992b-b9cfdc22b32f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.484873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.485712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.485909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.486196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] PATH_INFO: `/v3/projects/a5628234021d4e2c9aea69638b52d2e2/users/3c54656e6d5a46e699ca2653ec8f93c5/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.486451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9598143d-4ce9-49e2-bce9-e82c0c0f3091 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.496224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.496929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.497280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.497622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.525040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.526839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.540876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5628234021d4e2c9aea69638b52d2e2, user_id=3c54656e6d5a46e699ca2653ec8f93c5, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.547953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.564176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 832/1641] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:54.581227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 810/1642] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:54 2022] PUT /identity/v3/projects/a5628234021d4e2c9aea69638b52d2e2/users/3c54656e6d5a46e699ca2653ec8f93c5/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:54.586311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:54.602778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f217ba46-e479-4b4a-a02e-a11d6a4a27e8 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.636319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:54.756982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.757729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.758294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.758578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.759467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.761356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.770150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defb8e45-a4be-4087-a192-1862989974f2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.780926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd0f279-cd07-4880-b6ce-023715f1b1cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.781567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd0f279-cd07-4880-b6ce-023715f1b1cb service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.781845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd0f279-cd07-4880-b6ce-023715f1b1cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.782115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd0f279-cd07-4880-b6ce-023715f1b1cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.784896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 833/1643] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:54 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:54.789168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:54.811376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c98cb-8351-4ce8-854a-db5475aace8b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:54.826511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c8bf6-75b3-4dea-a982-ec526b26350a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:54.828605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c8bf6-75b3-4dea-a982-ec526b26350a admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.829307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c8bf6-75b3-4dea-a982-ec526b26350a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.831124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c8bf6-75b3-4dea-a982-ec526b26350a admin admin] PATH_INFO: `/v3/projects/a5628234021d4e2c9aea69638b52d2e2/users/3c54656e6d5a46e699ca2653ec8f93c5/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.836066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96c8bf6-75b3-4dea-a982-ec526b26350a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5628234021d4e2c9aea69638b52d2e2, user_id=3c54656e6d5a46e699ca2653ec8f93c5, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.844416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96c8bf6-75b3-4dea-a982-ec526b26350a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.875810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 834/1644] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:54 2022] PUT /identity/v3/projects/a5628234021d4e2c9aea69638b52d2e2/users/3c54656e6d5a46e699ca2653ec8f93c5/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:54.893703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.894467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.894975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.960419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd0f279-cd07-4880-b6ce-023715f1b1cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:54.962538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd0f279-cd07-4880-b6ce-023715f1b1cb service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:54.972657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] MFA Rules not processed for user `3c54656e6d5a46e699ca2653ec8f93c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:54.974479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:54.976482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 811/1645] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:54 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:54.977985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4828fc5-1bc4-41d2-9606-05e2d22137fa None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.978268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4828fc5-1bc4-41d2-9606-05e2d22137fa None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.980740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4828fc5-1bc4-41d2-9606-05e2d22137fa None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:54.980740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 812/1646] 10.222.0.20 () {54 vars in 912 bytes} [Fri Sep 9 04:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:38:54.986865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:54.987257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:54.987543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:55.027375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:55.028943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:55.072811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:55.143530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 835/1647] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:54 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:55.157617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:55.226466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:55.239002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 813/1648] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:38:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:55.279326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef436acd-dae4-48a3-ba5c-82c03b25feb3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:55.285907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:55.288281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:55.288281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:55.288281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:55.300537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:55.303567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:55.328114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 836/1649] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:55 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:55.384527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:55.403871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abdffa-34cf-4168-ae1d-e3ebfbc07037 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:55.424698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e70f4ec-6325-4f00-960c-ed9a93dec60a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:55.425343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e70f4ec-6325-4f00-960c-ed9a93dec60a service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:55.425630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e70f4ec-6325-4f00-960c-ed9a93dec60a service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:55.476769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e70f4ec-6325-4f00-960c-ed9a93dec60a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:55.476769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e70f4ec-6325-4f00-960c-ed9a93dec60a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:55.476769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e70f4ec-6325-4f00-960c-ed9a93dec60a service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:55.494575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 814/1650] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:55.697194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:55.869872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c955b3f6-fc48-4953-8740-e14a0b4803a8 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:55.879024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:55.879688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:55.879972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:55.880252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] PATH_INFO: `/v3/users/7169599dbb3e47f098cacbfe1ae5bfcc` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:55.885105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7169599dbb3e47f098cacbfe1ae5bfcc)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:55.889712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:55.999740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7169599dbb3e47f098cacbfe1ae5bfcc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:56.018184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7169599dbb3e47f098cacbfe1ae5bfcc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:56.047587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7169599dbb3e47f098cacbfe1ae5bfcc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:56.049977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7169599dbb3e47f098cacbfe1ae5bfcc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:56.075810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7169599dbb3e47f098cacbfe1ae5bfcc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:56.094189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 837/1651] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:38:55 2022] DELETE /identity/v3/users/7169599dbb3e47f098cacbfe1ae5bfcc => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:56.098776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:56.099536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:56.099963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:56.155428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:56.156680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:56.272482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 815/1652] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:56.370767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:56.387788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:56.438574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:56.440114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:56.642659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451ef8db-0065-4ab6-aa28-5db69de302a0 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:56.643473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935b11c6-419d-4b00-b63b-c12fab56073f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:56.653702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f21354-c08f-4d5e-bd9b-2dced446fea6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:56.654485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f21354-c08f-4d5e-bd9b-2dced446fea6 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:56.654812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f21354-c08f-4d5e-bd9b-2dced446fea6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:56.655198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f21354-c08f-4d5e-bd9b-2dced446fea6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:56.665983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:56.666724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:56.667245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:56.667621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:56.672709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:56.687714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:56.695817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f21354-c08f-4d5e-bd9b-2dced446fea6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:56.699408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f21354-c08f-4d5e-bd9b-2dced446fea6 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:56.738911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 838/1653] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:38:56.810167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:56.815840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:56.817570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:56.834832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 816/1654] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:38:56 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:38:56.962814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:56.964754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:56.966537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:57.037037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] MFA Rules not processed for user `225e6103cfc144edbc304779e2878f82`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:57.038326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:57.235025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:57.236328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 839/1655] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:56 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:57.237418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:57.271014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acdcbeb-ceb0-4d9a-9859-55d287d77a61 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:57.290158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:57.292301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:57.292770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:57.295126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:57.298117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:57.307079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:57.436940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:57.437885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 817/1656] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:57 2022] POST /identity/v3/projects => generated 403 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:38:57.468473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950d480-dc20-4c35-b3a4-064398d8c050 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:57.617390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:57.618345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:57.618624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:57.618901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:57.619659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:57.621933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:57.707805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 840/1657] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:57 2022] POST /identity/v3/users => generated 308 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:57.712176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:57.726299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fde52b8-5280-478a-af1b-500186a565ec admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:57.747221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:57.747867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:57.748143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:57.748425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:57.749089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:57.767561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:57.786623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 818/1658] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:57 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:57.790167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:57.809284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d913b0ae-f124-4f81-a652-00404e693ff3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:57.844751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:57.845552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:57.847280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:57.847982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] PATH_INFO: `/v3/projects/6416a2d3859545e39eacea132819e4e3/users/1ec00fb6e8e8491b8f17c3ce87866d06/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:57.903344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6416a2d3859545e39eacea132819e4e3, user_id=1ec00fb6e8e8491b8f17c3ce87866d06, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:57.908405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:57.941813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 841/1659] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:57 2022] PUT /identity/v3/projects/6416a2d3859545e39eacea132819e4e3/users/1ec00fb6e8e8491b8f17c3ce87866d06/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:57.957086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:57.993143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f29955d-d22c-4179-9aa4-9d8616fd0dd2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:58.161340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:58.163783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:58.164176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:58.164434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:58.165305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:58.167373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:58.198468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 819/1660] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:57 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:58.202697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:58.221275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8814253-a2f4-4c94-a0fe-7fcf2ccb0c65 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:58.232846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8ae3ad-cf07-4d00-a9da-b1e07f11dc8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:58.233555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ae3ad-cf07-4d00-a9da-b1e07f11dc8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:58.233827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ae3ad-cf07-4d00-a9da-b1e07f11dc8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:58.235849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ae3ad-cf07-4d00-a9da-b1e07f11dc8b admin admin] PATH_INFO: `/v3/projects/6416a2d3859545e39eacea132819e4e3/users/1ec00fb6e8e8491b8f17c3ce87866d06/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:58.239058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ae3ad-cf07-4d00-a9da-b1e07f11dc8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=6416a2d3859545e39eacea132819e4e3, user_id=1ec00fb6e8e8491b8f17c3ce87866d06, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:58.241784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ae3ad-cf07-4d00-a9da-b1e07f11dc8b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:58.275112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 842/1661] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:38:58 2022] PUT /identity/v3/projects/6416a2d3859545e39eacea132819e4e3/users/1ec00fb6e8e8491b8f17c3ce87866d06/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:58.863126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:58.868551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:58.948243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dd0d3-3f80-4004-b85b-7f9de5802277 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:59.035614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:59.036424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.037315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.037955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] PATH_INFO: `/v3/projects/bb415f49dfd64fdc88c46543878c6e74` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.039587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb415f49dfd64fdc88c46543878c6e74)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:59.041761 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:59.152740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb415f49dfd64fdc88c46543878c6e74', 'request_id': 'req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:59.172141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb415f49dfd64fdc88c46543878c6e74', 'request_id': 'req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:59.174905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:59.183735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb415f49dfd64fdc88c46543878c6e74', 'request_id': 'req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:59.228324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] The token cache is being invalidate because project bb415f49dfd64fdc88c46543878c6e74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:38:59.228622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:59.230422 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b0bfa6eb-480b-4998-9fdb-bfaeedcad5fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:38:59.241593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 820/1662] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:38:58 2022] DELETE /identity/v3/projects/bb415f49dfd64fdc88c46543878c6e74 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:38:59.246454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.247040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.247119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.298037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:59.299470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:59.381008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:38:59.455243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 843/1663] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:38:59.471250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.471250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.471250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.496665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:38:59.498211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:38:59.508365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 821/1664] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:38:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:38:59.512316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:59.531253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa5fffd-7b19-4b94-90b4-f5c85903a5d9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:59.543824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:59.544470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.544750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.545026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.545691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:38:59.545996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:59.552096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:59.588590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 844/1665] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:38:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:38:59.593740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:59.608053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1551808-9220-47e7-bb5c-bb0e8d30a1bb None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:59.620412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:59.621047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.621333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.621697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.626861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:59.629255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:59.696154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 822/1666] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:38:59 2022] POST /identity/v3/projects => generated 381 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:38:59.700063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:59.714765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd11128-3c7b-41b2-8542-14c809170a82 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:59.870213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:59.873098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.873267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.873559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.875375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:59.880340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:59.929095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 845/1667] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:38:59 2022] POST /identity/v3/users => generated 291 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:38:59.932398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:38:59.949381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f5c6e3-9bf9-4906-b6ef-77445d948ac5 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:38:59.959455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:38:59.962980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:38:59.966607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:38:59.966607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:38:59.967512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:38:59.970074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:38:59.994515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 823/1668] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:59 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:38:59.999049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:00.015215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815e811e-5400-48b7-8169-b93e86d19136 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:00.028666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f2aa8-baf6-4f68-be6a-12bfd0e0a2aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:00.029460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f2aa8-baf6-4f68-be6a-12bfd0e0a2aa admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:00.029836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f2aa8-baf6-4f68-be6a-12bfd0e0a2aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:00.030206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f2aa8-baf6-4f68-be6a-12bfd0e0a2aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:00.031018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7f2aa8-baf6-4f68-be6a-12bfd0e0a2aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:00.032679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7f2aa8-baf6-4f68-be6a-12bfd0e0a2aa admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:00.060723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 846/1669] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:38:59 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:00.062638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:00.079347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0702de-ca94-41d1-b06f-c2234d06f9f1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:00.091656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:00.092745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:00.093428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:00.093764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] PATH_INFO: `/v3/projects/32ecb8ce7f074f8389672f65f860f5a2/users/2954414d7c0f4e14a48f9951b5f3a06c/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:00.127278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32ecb8ce7f074f8389672f65f860f5a2, user_id=2954414d7c0f4e14a48f9951b5f3a06c, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:00.131282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:00.169012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 824/1670] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:00 2022] PUT /identity/v3/projects/32ecb8ce7f074f8389672f65f860f5a2/users/2954414d7c0f4e14a48f9951b5f3a06c/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:00.184630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2caa40-1741-418e-a438-de815e905e3b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:00.185218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2caa40-1741-418e-a438-de815e905e3b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:00.185677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2caa40-1741-418e-a438-de815e905e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:00.250289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7c2caa40-1741-418e-a438-de815e905e3b None None] MFA Rules not processed for user `2954414d7c0f4e14a48f9951b5f3a06c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:00.252261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7c2caa40-1741-418e-a438-de815e905e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:00.411918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 847/1671] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:00.416413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:00.511901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:00.512195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:00.512497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:00.581511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] MFA Rules not processed for user `1ec00fb6e8e8491b8f17c3ce87866d06`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:00.583753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:00.656300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e1792-1863-42f9-870a-a4aff5c83c19 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:00.667846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:00.668729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:00.669005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:00.669282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:00.670027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:00.678892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:00.679592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:00.748714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 825/1672] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:00 2022] POST /identity/v3/projects => generated 381 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:00.749656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:00.909331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 848/1673] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:00 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 397 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:00.934862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9c8b5f-940c-4133-994b-c6d55b349052 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:00.954549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:00.957050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:00.957644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:00.958067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:00.960950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:00.963447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:01.031014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=93,keystone:DELETE=1,keystone:INSERT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:01.065794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 826/1674] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:00 2022] POST /identity/v3/users => generated 297 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:01.076833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:01.103946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056d82d-0a77-4785-b932-f38d624789d1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:01.116447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:01.117524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:01.118134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:01.118564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:01.119439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:01.121567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:01.146789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 849/1675] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:01 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:01.150001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:01.164489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a5be3e-a3b6-4f44-aa4e-79a3441592f1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:01.178124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:01.178873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:01.179122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:01.179457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] PATH_INFO: `/v3/projects/0f44ee8cb38e434983339d605b1121c6/users/542f7643ffab422183f877ef7e28743d/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:01.216942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f44ee8cb38e434983339d605b1121c6, user_id=542f7643ffab422183f877ef7e28743d, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:01.229707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:01.264207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 827/1676] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:01 2022] PUT /identity/v3/projects/0f44ee8cb38e434983339d605b1121c6/users/542f7643ffab422183f877ef7e28743d/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:01.264495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:01.282139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09473484-09f7-43b6-a505-d7d1fb6c217d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:01.457661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872f8898-84c9-4c2c-8890-9721c1b692f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:01.459409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872f8898-84c9-4c2c-8890-9721c1b692f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:01.460185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872f8898-84c9-4c2c-8890-9721c1b692f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:01.461177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872f8898-84c9-4c2c-8890-9721c1b692f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:01.462598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872f8898-84c9-4c2c-8890-9721c1b692f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:01.465319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872f8898-84c9-4c2c-8890-9721c1b692f7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:01.477810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 850/1677] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:01 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:01.483202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:01.510884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdd86f7-0acb-49fb-9bdc-ca83ce018720 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:01.524435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:01.525515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:01.525515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:01.525721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] PATH_INFO: `/v3/projects/0f44ee8cb38e434983339d605b1121c6/users/542f7643ffab422183f877ef7e28743d/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:01.528108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f44ee8cb38e434983339d605b1121c6, user_id=542f7643ffab422183f877ef7e28743d, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:01.537996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:01.588065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 828/1678] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:01 2022] PUT /identity/v3/projects/0f44ee8cb38e434983339d605b1121c6/users/542f7643ffab422183f877ef7e28743d/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:01.592470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:01.592866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:01.593220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:01.667237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] MFA Rules not processed for user `542f7643ffab422183f877ef7e28743d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:01.670407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:01.888218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 851/1679] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:01 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:01.901588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:01.903526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:02.059818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e382a58-3cfd-4c75-8487-390fbffa9fc3 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:02.065847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4ddb43-48ef-4bea-8682-5b3597ec540e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:02.066638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4ddb43-48ef-4bea-8682-5b3597ec540e service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:02.067821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4ddb43-48ef-4bea-8682-5b3597ec540e service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:02.068736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4ddb43-48ef-4bea-8682-5b3597ec540e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:02.086987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4ddb43-48ef-4bea-8682-5b3597ec540e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:02.091670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4ddb43-48ef-4bea-8682-5b3597ec540e service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:02.150908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 829/1680] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:01 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:02.368423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:02.370693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:02.405165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f440fe-40f0-4d42-a68d-2b00042485ca None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:02.414813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94746e1-e99a-4bda-a843-80b5ab281d28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:02.415806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94746e1-e99a-4bda-a843-80b5ab281d28 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:02.417271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94746e1-e99a-4bda-a843-80b5ab281d28 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:02.419216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94746e1-e99a-4bda-a843-80b5ab281d28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:02.497690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:02.566507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94746e1-e99a-4bda-a843-80b5ab281d28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:02.569717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94746e1-e99a-4bda-a843-80b5ab281d28 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:02.572322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:02.573289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:02.573646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:02.600496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 852/1681] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:02 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:02.662583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] MFA Rules not processed for user `d3c29c23d7524adf9c5a3271224ac8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:02.664013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:02.866538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d84300-93be-47bf-b08b-925147304738 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:02.867207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d84300-93be-47bf-b08b-925147304738 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:02.867716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d84300-93be-47bf-b08b-925147304738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:02.876008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 830/1682] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:02 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:02.906478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e1d84300-93be-47bf-b08b-925147304738 None None] MFA Rules not processed for user `d3c29c23d7524adf9c5a3271224ac8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:02.907458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e1d84300-93be-47bf-b08b-925147304738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:02.924923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 853/1683] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:02 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:02.939214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:02.941576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:03.064391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:03.105872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2034b08a-d619-4aba-8b1b-4682e7a9b32e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:03.112126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0674083-63c5-41a2-a229-bce53904df45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:03.113707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0674083-63c5-41a2-a229-bce53904df45 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:03.114581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0674083-63c5-41a2-a229-bce53904df45 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:03.115721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0674083-63c5-41a2-a229-bce53904df45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:03.125292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0674083-63c5-41a2-a229-bce53904df45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:03.127838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0674083-63c5-41a2-a229-bce53904df45 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:03.151178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 831/1684] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:02 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:03.393918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe7c219-2860-4049-b833-55b62ad22abe None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:03.394363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe7c219-2860-4049-b833-55b62ad22abe None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:03.394596 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe7c219-2860-4049-b833-55b62ad22abe None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:03.396852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 854/1685] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:39:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:39:03.428036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacbf36-cfa1-497d-a70f-cd3b1805accf None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:03.428586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacbf36-cfa1-497d-a70f-cd3b1805accf None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:03.428901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacbf36-cfa1-497d-a70f-cd3b1805accf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:03.527287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ffacbf36-cfa1-497d-a70f-cd3b1805accf None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:03.530990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ffacbf36-cfa1-497d-a70f-cd3b1805accf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:03.760125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 832/1686] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:39:03 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:03.914424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe7c219-2860-4049-b833-55b62ad22abe None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:03.916751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cfe7c219-2860-4049-b833-55b62ad22abe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:03.956321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe7c219-2860-4049-b833-55b62ad22abe None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:03.985045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7407d1b4-2738-42c1-bb8b-2557c1f4b97e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:03.985675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7407d1b4-2738-42c1-bb8b-2557c1f4b97e service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:03.987416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7407d1b4-2738-42c1-bb8b-2557c1f4b97e service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:03.987416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7407d1b4-2738-42c1-bb8b-2557c1f4b97e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:04.006076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7407d1b4-2738-42c1-bb8b-2557c1f4b97e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:04.009355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7407d1b4-2738-42c1-bb8b-2557c1f4b97e service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:04.034225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 855/1687] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:04.117150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ca957-d889-434d-9a7c-b10fa2a96f39 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:04.117644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ca957-d889-434d-9a7c-b10fa2a96f39 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:04.117719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ca957-d889-434d-9a7c-b10fa2a96f39 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:04.120506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 833/1688] 10.222.0.20 () {54 vars in 912 bytes} [Fri Sep 9 04:39:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 09 04:39:04.127048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:04.127340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:04.127624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:04.251235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:04.252583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:04.429296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 856/1689] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:39:04 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 303 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:04.442072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5ca957-d889-434d-9a7c-b10fa2a96f39 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:04.445403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7c5ca957-d889-434d-9a7c-b10fa2a96f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:04.467785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5ca957-d889-434d-9a7c-b10fa2a96f39 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:04.491074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:04.491770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:04.492073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:04.492503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:04.504695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:04.507248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:04.550265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 834/1690] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:05.767625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:05.780429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:05.787380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a101ff-6ffb-4577-8004-6e2d4d309e2e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:05.799833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fbb5b0-3738-4680-81ac-55c8273df4d1 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:05.810428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:05.812160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:05.812933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:05.813372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:05.813961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:05.814181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:05.814783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:05.815187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:05.820550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:05.828053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:05.859922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:05.868061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7599678e-3617-4ca8-8650-0f03ecb816ed service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:05.913552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 857/1691] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:05.961891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:05.963937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:05.987392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 835/1692] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:05 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:06.054610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:06.054828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:06.055202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:06.129802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] MFA Rules not processed for user `d3c29c23d7524adf9c5a3271224ac8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:06.136420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:06.166695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 858/1693] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:06.759535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:06.761280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:07.033561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f68afa5-1c40-41dc-b64f-bed570f9d0a8 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:07.058425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:07.059139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:07.059407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:07.059682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] PATH_INFO: `/v3/users/225e6103cfc144edbc304779e2878f82` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:07.061942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] RBAC: Authorizing `identity:delete_user(user_id=225e6103cfc144edbc304779e2878f82)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:07.065195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: WARNING py.warnings [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:39:07.065195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: warnings.warn(msg) Sep 09 04:39:07.065195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: Sep 09 04:39:07.071350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:07.080554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:07.297078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '225e6103cfc144edbc304779e2878f82'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:07.326352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '225e6103cfc144edbc304779e2878f82'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:07.416046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '225e6103cfc144edbc304779e2878f82'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:07.417325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '225e6103cfc144edbc304779e2878f82'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:07.454554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '225e6103cfc144edbc304779e2878f82'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:07.496197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 836/1694] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:06 2022] DELETE /identity/v3/users/225e6103cfc144edbc304779e2878f82 => generated 0 bytes in 737 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:08.108381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:08.135693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c19ca-c671-426b-b449-6dc27a91a918 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:08.397294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:08.397953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:08.398240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:08.398520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] PATH_INFO: `/v3/projects/6dd9e6c08659499c864d7f799f07fa78` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:08.403266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6dd9e6c08659499c864d7f799f07fa78)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:08.404330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 09 04:39:08.404330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Sep 09 04:39:08.404330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: Sep 09 04:39:08.405590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:08.546242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:08.618638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6dd9e6c08659499c864d7f799f07fa78', 'request_id': 'req-6832c81d-228c-4f82-b795-becf9ba5ec9f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:08.642062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6dd9e6c08659499c864d7f799f07fa78', 'request_id': 'req-6832c81d-228c-4f82-b795-becf9ba5ec9f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:08.653132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6dd9e6c08659499c864d7f799f07fa78', 'request_id': 'req-6832c81d-228c-4f82-b795-becf9ba5ec9f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:08.712846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] The token cache is being invalidate because project 6dd9e6c08659499c864d7f799f07fa78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:08.713575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:08.715042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:08.719453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 859/1695] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:08 2022] DELETE /identity/v3/projects/6dd9e6c08659499c864d7f799f07fa78 => generated 0 bytes in 611 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:08.723479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:08.726119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:08.754835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb3717-5e78-43b9-8b01-a61bf85e794b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:08.885679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:08.886399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:08.886691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:08.886990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] PATH_INFO: `/v3/users/1ec00fb6e8e8491b8f17c3ce87866d06` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:08.888575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ec00fb6e8e8491b8f17c3ce87866d06)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:08.891016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:08.979266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ec00fb6e8e8491b8f17c3ce87866d06'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:08.995940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ec00fb6e8e8491b8f17c3ce87866d06'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.029843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ec00fb6e8e8491b8f17c3ce87866d06'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.030967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ec00fb6e8e8491b8f17c3ce87866d06'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.050200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-bf21c6c5-ea66-406f-bd8e-88278dc65af1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ec00fb6e8e8491b8f17c3ce87866d06'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.052281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:09.064988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 837/1696] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:08 2022] DELETE /identity/v3/users/1ec00fb6e8e8491b8f17c3ce87866d06 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:09.607509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:09.609271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:09.628387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6832c81d-228c-4f82-b795-becf9ba5ec9f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:09.698391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:09.699260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:09.699736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:09.700016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] PATH_INFO: `/v3/projects/6416a2d3859545e39eacea132819e4e3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:09.701328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6416a2d3859545e39eacea132819e4e3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:09.703526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:09.767675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6416a2d3859545e39eacea132819e4e3', 'request_id': 'req-3abd6c7e-5063-47cf-9269-800bad08fec3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.779564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6416a2d3859545e39eacea132819e4e3', 'request_id': 'req-3abd6c7e-5063-47cf-9269-800bad08fec3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.787331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6416a2d3859545e39eacea132819e4e3', 'request_id': 'req-3abd6c7e-5063-47cf-9269-800bad08fec3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.814118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] The token cache is being invalidate because project 6416a2d3859545e39eacea132819e4e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:09.814413 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.815488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3abd6c7e-5063-47cf-9269-800bad08fec3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:09.821167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 860/1697] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:09 2022] DELETE /identity/v3/projects/6416a2d3859545e39eacea132819e4e3 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:09.822888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:09.823362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:09.823457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:09.845421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:09.847117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:09.939284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 838/1698] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:09.948926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:09.949201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:09.949483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:09.973273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:09.975839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:09.986513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 861/1699] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:09.991574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.003567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b5db73-920c-493c-9679-5149b1c55269 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.009013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.009592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.009878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.010160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.011015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:10.011341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.013167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408c6973-2002-4826-a732-0cb37b614b9d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:10.026988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 839/1700] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:10.030681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.041772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187c92a-88f4-4501-8daf-5fa653e53667 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.051681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.051988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.052780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.057419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.067889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.068727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.068945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.069617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.072831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:10.090342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] MFA Rules not processed for user `befee01f638c4e7681b30c64acad6a6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:10.091648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:10.161336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 862/1701] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:10 2022] POST /identity/v3/projects => generated 401 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:10.167279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.214812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:10.288548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 840/1702] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:10 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:10.296910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.298653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:10.348798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a771b-994f-40ab-928b-447817ca9ec0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.363954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.365294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.365759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.366041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.366692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.368744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:10.491102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 863/1703] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:10 2022] POST /identity/v3/users => generated 301 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:10.495264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.535033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efbc884-249b-46c3-a3ce-9ef34b6afa72 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.559699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.560710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.561089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.562217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.563037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.564557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a3d3c5-9ad9-458f-a7f6-b259a59f0c38 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.573870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:10.600314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:10.615244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 864/1704] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:10.619354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.643866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c91618-cf06-4fc1-b789-7ece3c728ef2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.659805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.661647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.663607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.664156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.665209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.667390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:10.681456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 865/1705] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:10.686529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:10.714500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b3b4b-6215-473a-b52e-72807f71f94b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:10.724925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb3bc9-765c-4946-86b2-a4ea3e576826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.725616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb3bc9-765c-4946-86b2-a4ea3e576826 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.725894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb3bc9-765c-4946-86b2-a4ea3e576826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.726173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb3bc9-765c-4946-86b2-a4ea3e576826 admin admin] PATH_INFO: `/v3/projects/4305282983c649c3bbabec098e027ca5/users/d375ac46c0ab429ca6a1d18b3e3fd4f9/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.759479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eb3bc9-765c-4946-86b2-a4ea3e576826 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4305282983c649c3bbabec098e027ca5, user_id=d375ac46c0ab429ca6a1d18b3e3fd4f9, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.761952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eb3bc9-765c-4946-86b2-a4ea3e576826 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:10.763327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:10.764185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.767653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.768259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.820010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 866/1706] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:10 2022] PUT /identity/v3/projects/4305282983c649c3bbabec098e027ca5/users/d375ac46c0ab429ca6a1d18b3e3fd4f9/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:10.821518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:10.823818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:10.824106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:10.892529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] MFA Rules not processed for user `d375ac46c0ab429ca6a1d18b3e3fd4f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:10.900378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:10.957024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:10.959388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:11.011373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 841/1707] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:10 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 715 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:11.047231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:11.047371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 867/1708] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:10 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:11.064023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a65dc2-d342-4937-9104-8bd546a58eb8 service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:11.223760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376edb5a-db0d-4d2b-9190-84f3569aa6be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:11.226192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376edb5a-db0d-4d2b-9190-84f3569aa6be admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:11.226614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376edb5a-db0d-4d2b-9190-84f3569aa6be admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:11.227048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376edb5a-db0d-4d2b-9190-84f3569aa6be admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:11.229169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376edb5a-db0d-4d2b-9190-84f3569aa6be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:11.236191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376edb5a-db0d-4d2b-9190-84f3569aa6be admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:11.330158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 842/1709] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:11 2022] POST /identity/v3/projects => generated 401 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:11.334497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:11.348852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48767d74-5e52-4072-a7af-1e119e810e5c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:11.367846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003153cb-a6b7-40e7-9643-b1d5f6f716fd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:11.368315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003153cb-a6b7-40e7-9643-b1d5f6f716fd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:11.368689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003153cb-a6b7-40e7-9643-b1d5f6f716fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:11.437944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-003153cb-a6b7-40e7-9643-b1d5f6f716fd None None] MFA Rules not processed for user `161e111ce6384c0caadaa06d4bcffee2`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:11.439921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-003153cb-a6b7-40e7-9643-b1d5f6f716fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:11.479214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:11.517028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:11.537495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:11.548398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:11.549295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:11.550377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:11.551960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:11.558806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:11.646355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 868/1710] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:11 2022] POST /identity/v3/users => generated 307 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:11.649385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:11.705053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 843/1711] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:11.716392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:11.716881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:11.717498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:11.719800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df66df90-bd84-4ec5-875f-d17aa0437f28 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:11.755768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:11.757194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:11.758967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:11.759402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:11.765301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:11.770895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:11.780780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] MFA Rules not processed for user `161e111ce6384c0caadaa06d4bcffee2`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:11.789086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:11.804224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 844/1712] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:11.815712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:11.853314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 869/1713] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:11 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:11.858761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:11.923746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bbbbcc-d24f-4431-9ea9-61b8076d05f0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:11.941304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:11.941993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:11.942278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:11.942709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] PATH_INFO: `/v3/projects/7b909cddd2404bb18989d4c1f1d2a21a/users/a7cf6befca02444a84f4843d0c5fbfb1/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:12.009195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b909cddd2404bb18989d4c1f1d2a21a, user_id=a7cf6befca02444a84f4843d0c5fbfb1, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:12.012163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:12.061611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 870/1714] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:11 2022] PUT /identity/v3/projects/7b909cddd2404bb18989d4c1f1d2a21a/users/a7cf6befca02444a84f4843d0c5fbfb1/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:12.062584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:12.107722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6394e8c-b477-470a-b465-77a7683b6c1b None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:12.338573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e2501d-4f0f-4ee3-964d-1d39c71e1cc7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:12.366067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:12.366707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:12.367034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:12.369290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:12.370822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:12.375635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:12.405632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 871/1715] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:12.412667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:12.438740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ec7942-afee-45b4-bb6e-c24aac7361ed admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:12.459851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c2f787-d254-4add-a115-4c8c3527e5d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:12.462562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2f787-d254-4add-a115-4c8c3527e5d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:12.463092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2f787-d254-4add-a115-4c8c3527e5d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:12.463519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2f787-d254-4add-a115-4c8c3527e5d7 admin admin] PATH_INFO: `/v3/projects/7b909cddd2404bb18989d4c1f1d2a21a/users/a7cf6befca02444a84f4843d0c5fbfb1/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:12.467737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c2f787-d254-4add-a115-4c8c3527e5d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b909cddd2404bb18989d4c1f1d2a21a, user_id=a7cf6befca02444a84f4843d0c5fbfb1, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:12.468641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:12.473498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:12.474022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:12.474449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:12.479422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c2f787-d254-4add-a115-4c8c3527e5d7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:12.532393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 872/1716] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:12 2022] PUT /identity/v3/projects/7b909cddd2404bb18989d4c1f1d2a21a/users/a7cf6befca02444a84f4843d0c5fbfb1/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:12.543101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e76594-62b5-4f84-8c6e-92d728acc070 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:12.543377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e76594-62b5-4f84-8c6e-92d728acc070 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:12.546110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e76594-62b5-4f84-8c6e-92d728acc070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:12.660603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:12.672645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:12.673436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-12e76594-62b5-4f84-8c6e-92d728acc070 None None] MFA Rules not processed for user `a7cf6befca02444a84f4843d0c5fbfb1`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:12.674260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:12.675486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12e76594-62b5-4f84-8c6e-92d728acc070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:12.715820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 845/1717] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 901 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:12.790699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:12.792213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:12.823578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ee70ce-57fb-4088-af0a-9f3129d0ce02 service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:12.939158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 873/1718] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:12 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:12.939158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be25656-abb7-423d-a227-38f80ffd73a8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:12.939158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be25656-abb7-423d-a227-38f80ffd73a8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:12.939158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be25656-abb7-423d-a227-38f80ffd73a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:12.996216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:12.999197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:12.999921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:13.000651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] PATH_INFO: `/v3/users/b2c41a8904124aa6b31ae272a684b060` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:13.003027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2c41a8904124aa6b31ae272a684b060)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:13.009197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:13.024402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:13.048902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0be25656-abb7-423d-a227-38f80ffd73a8 None None] MFA Rules not processed for user `bf68e214fd4740be96f2d6c4325f6ee9`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:13.053883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0be25656-abb7-423d-a227-38f80ffd73a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:13.128166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2c41a8904124aa6b31ae272a684b060'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:13.146881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2c41a8904124aa6b31ae272a684b060'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:13.186279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2c41a8904124aa6b31ae272a684b060'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:13.187286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2c41a8904124aa6b31ae272a684b060'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:13.204640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2c41a8904124aa6b31ae272a684b060'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:13.231365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 846/1719] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:12 2022] DELETE /identity/v3/users/b2c41a8904124aa6b31ae272a684b060 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:13.232845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:13.234429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:13.310390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 874/1720] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 365 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:13.310390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7ce33-ba39-464d-9671-d92a8ef60871 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:13.310390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7ce33-ba39-464d-9671-d92a8ef60871 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:13.310390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7ce33-ba39-464d-9671-d92a8ef60871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:13.409308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5da344-c1ba-4978-882e-e20bee134898 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:13.422237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:13.422237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:13.422237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:13.422237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:13.429219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:13.479489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e0e7ce33-ba39-464d-9671-d92a8ef60871 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:13.480765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e0e7ce33-ba39-464d-9671-d92a8ef60871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:13.504107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:13.632249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:13.635269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:13.676952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 875/1721] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:39:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 375 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:13.678653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:13.679025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:13.679265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:13.693876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 847/1722] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:13 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:13.696218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:13.699246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:13.747646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:13.754955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:14.019162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d852063-c645-41f1-a78d-7f9def5261bb service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:14.031190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 876/1723] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:14.034651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:14.038968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:14.039375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:14.039630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:14.040076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:14.074538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36a94db-9a16-4876-9aad-f07d755d1fca None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:14.085242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:14.088126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:14.088661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:14.089325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:14.094443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:14.145604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:14.229481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:14.231262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:14.263447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 848/1724] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 567 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:14.264202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:14.269798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:14.373459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:14.374924 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:14.401075 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:14.415181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 877/1725] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:14 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:14.433812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:14.437214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:14.477286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d10155-eed1-4b3d-96ab-1b5aca584bd8 service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:14.491904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:14.492540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:14.492820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:14.493716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:14.512206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:14.517033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:14.551220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 849/1726] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:39:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:39:14.653468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5fca8f-6322-421a-9549-a4423464b26d service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:14.676580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8adf9f-2583-48ce-ae49-b940e309e023 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:14.677848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8adf9f-2583-48ce-ae49-b940e309e023 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:14.678454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8adf9f-2583-48ce-ae49-b940e309e023 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:14.678454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8adf9f-2583-48ce-ae49-b940e309e023 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:14.740417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8adf9f-2583-48ce-ae49-b940e309e023 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:14.742170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8adf9f-2583-48ce-ae49-b940e309e023 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:14.770018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 878/1727] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:15.607550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:15.609612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:15.676717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66da3b00-3ec3-47d7-8e72-725319bc8d12 service swift] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:15.711700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:15.713543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:15.713917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:15.714316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] PATH_INFO: `/v3/users/2954414d7c0f4e14a48f9951b5f3a06c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:15.718256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2954414d7c0f4e14a48f9951b5f3a06c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:15.723325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:15.894147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2954414d7c0f4e14a48f9951b5f3a06c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:15.914182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2954414d7c0f4e14a48f9951b5f3a06c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:15.986875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2954414d7c0f4e14a48f9951b5f3a06c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:15.994243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2954414d7c0f4e14a48f9951b5f3a06c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:16.027026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d1962cb8-61b9-45c9-ad4e-b6db0a7fa754 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2954414d7c0f4e14a48f9951b5f3a06c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:16.056971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 850/1728] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:15 2022] DELETE /identity/v3/users/2954414d7c0f4e14a48f9951b5f3a06c => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:16.061698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:16.062565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:16.063348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:16.128459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:16.131726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:16.426756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 879/1729] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:17.072882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:17.074082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:17.074546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:17.145089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:17.154796 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:17.158074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:17.159834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:17.440053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9e44c0-d2ce-4fd4-b4fa-13516c0f26ae None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:17.448534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 851/1730] 10.222.0.17 () {58 vars in 1015 bytes} [Fri Sep 9 04:39:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 377 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:17.453190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:17.459143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:17.468341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:17.468937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:17.505303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:17.506956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:17.513614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:17.531926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4594ccc9-6b02-4bdf-9706-a55fe63af70c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:17.536802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:17.543839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:17.544530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:17.544807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:17.545080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:17.556300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 880/1731] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:17.557475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:17.562228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:17.564903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:17.569059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:17.596572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0f8086-4a19-4185-ad29-830f3bf0f3c1 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:17.596836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 852/1732] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:17.603082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:17.604969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:17.605730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:17.606158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:17.617524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:17.620808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:17.636249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 881/1733] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:18.825879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:18.828600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:18.883756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68538c79-28e7-4340-8460-7756fb50654b service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:18.906849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:18.913573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:18.913931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:18.914746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] PATH_INFO: `/v3/projects/574802c84aa7424b9f4a819d8107a7a9` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:18.921370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=574802c84aa7424b9f4a819d8107a7a9)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:18.929720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:19.037808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:19.079800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '574802c84aa7424b9f4a819d8107a7a9', 'request_id': 'req-9eff1351-b91e-455a-b58b-ebd9582c0b67'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.129685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '574802c84aa7424b9f4a819d8107a7a9', 'request_id': 'req-9eff1351-b91e-455a-b58b-ebd9582c0b67'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.143432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '574802c84aa7424b9f4a819d8107a7a9', 'request_id': 'req-9eff1351-b91e-455a-b58b-ebd9582c0b67'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.173864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] The token cache is being invalidate because project 574802c84aa7424b9f4a819d8107a7a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:19.174501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.176112 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.185980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 853/1734] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:18 2022] DELETE /identity/v3/projects/574802c84aa7424b9f4a819d8107a7a9 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:19.186637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:19.210190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8c97c8-20b6-4535-b222-2da54155f685 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:19.378638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:19.379717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:19.381263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:19.381748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] PATH_INFO: `/v3/users/a71d2db2a2294af084c1fcca30624801` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:19.383744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] RBAC: Authorizing `identity:delete_user(user_id=a71d2db2a2294af084c1fcca30624801)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:19.387957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:19.537875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a71d2db2a2294af084c1fcca30624801'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.557687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a71d2db2a2294af084c1fcca30624801'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.623948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a71d2db2a2294af084c1fcca30624801'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.629392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a71d2db2a2294af084c1fcca30624801'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.668616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a71d2db2a2294af084c1fcca30624801'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:19.698161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 882/1735] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:19 2022] DELETE /identity/v3/users/a71d2db2a2294af084c1fcca30624801 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:20.233562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:20.235774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:20.272485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff1351-b91e-455a-b58b-ebd9582c0b67 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:20.513378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:20.514277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:20.514792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:20.515258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] PATH_INFO: `/v3/projects/32ecb8ce7f074f8389672f65f860f5a2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:20.516666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32ecb8ce7f074f8389672f65f860f5a2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:20.518899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:20.678285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:20.679901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:20.690060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32ecb8ce7f074f8389672f65f860f5a2', 'request_id': 'req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:20.709561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32ecb8ce7f074f8389672f65f860f5a2', 'request_id': 'req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:20.725219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32ecb8ce7f074f8389672f65f860f5a2', 'request_id': 'req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:20.725864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfc197c-5a7a-4ae8-9897-e602a47d3daf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:20.760342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] The token cache is being invalidate because project 32ecb8ce7f074f8389672f65f860f5a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:20.760648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:20.762713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:20.784008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 854/1736] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:20 2022] DELETE /identity/v3/projects/32ecb8ce7f074f8389672f65f860f5a2 => generated 0 bytes in 548 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:20.786708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:20.791776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:20.918594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:20.947625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:20.951294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:20.952894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:20.954127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] PATH_INFO: `/v3/users/d375ac46c0ab429ca6a1d18b3e3fd4f9` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:20.956975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d375ac46c0ab429ca6a1d18b3e3fd4f9)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:20.967493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:20.996255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:21.029633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0afa9a-e69e-4e32-95d5-56bf048a03e8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:21.041372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:21.042428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:21.042758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:21.043085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] PATH_INFO: `/v3/users/542f7643ffab422183f877ef7e28743d` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:21.044299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] RBAC: Authorizing `identity:delete_user(user_id=542f7643ffab422183f877ef7e28743d)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:21.049291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:21.194759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd375ac46c0ab429ca6a1d18b3e3fd4f9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.219051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd375ac46c0ab429ca6a1d18b3e3fd4f9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.219881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '542f7643ffab422183f877ef7e28743d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.263279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd375ac46c0ab429ca6a1d18b3e3fd4f9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.265543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd375ac46c0ab429ca6a1d18b3e3fd4f9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.280861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '542f7643ffab422183f877ef7e28743d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.288949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c537c1dd-1bd6-46e8-bdd7-e64a100bcf51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd375ac46c0ab429ca6a1d18b3e3fd4f9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.316357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 883/1737] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:20 2022] DELETE /identity/v3/users/d375ac46c0ab429ca6a1d18b3e3fd4f9 => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:21.318770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95059139-3021-4b30-886d-c9ded6c2199a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:21.319169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95059139-3021-4b30-886d-c9ded6c2199a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:21.321120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95059139-3021-4b30-886d-c9ded6c2199a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:21.354855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-95059139-3021-4b30-886d-c9ded6c2199a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:21.356210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-95059139-3021-4b30-886d-c9ded6c2199a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:21.361603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '542f7643ffab422183f877ef7e28743d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.366010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '542f7643ffab422183f877ef7e28743d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.391618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '542f7643ffab422183f877ef7e28743d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:21.409491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 855/1738] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:20 2022] DELETE /identity/v3/users/542f7643ffab422183f877ef7e28743d => generated 0 bytes in 621 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:21.623552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 884/1739] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:21.773478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:21.780298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:21.935680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c23b1a-58bc-4a43-8ca9-82093a25f855 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:21.941276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81571d93-6235-4660-9081-19f0496419b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:21.942662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81571d93-6235-4660-9081-19f0496419b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:21.943627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81571d93-6235-4660-9081-19f0496419b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:21.943910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81571d93-6235-4660-9081-19f0496419b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:22.208516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81571d93-6235-4660-9081-19f0496419b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:22.215552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81571d93-6235-4660-9081-19f0496419b8 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:22.254627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 856/1740] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:22.594220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9959afb-6105-4dd4-ba4f-8f0ae871940f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:22.594788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9959afb-6105-4dd4-ba4f-8f0ae871940f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:22.594788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9959afb-6105-4dd4-ba4f-8f0ae871940f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:22.600630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 885/1741] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:39:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:39:22.719638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:22.721134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:22.721608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:22.767837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:22.772336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:22.803634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:23.064726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 857/1742] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:39:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:23.292978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9959afb-6105-4dd4-ba4f-8f0ae871940f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:23.295962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f9959afb-6105-4dd4-ba4f-8f0ae871940f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:23.333747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9959afb-6105-4dd4-ba4f-8f0ae871940f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:23.344336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c624d79-2bda-4eec-b285-0104e1c247b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:23.345091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c624d79-2bda-4eec-b285-0104e1c247b2 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:23.345276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c624d79-2bda-4eec-b285-0104e1c247b2 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:23.345692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c624d79-2bda-4eec-b285-0104e1c247b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:23.373709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c624d79-2bda-4eec-b285-0104e1c247b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:23.377598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c624d79-2bda-4eec-b285-0104e1c247b2 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:23.427639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 886/1743] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:23.983370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:23.985360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:24.013998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c8862-2366-421d-a726-ddc9b4c98928 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:24.023454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fc08e-e934-419d-876a-224c75187b08 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:24.024001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fc08e-e934-419d-876a-224c75187b08 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:24.024001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fc08e-e934-419d-876a-224c75187b08 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:24.026498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 887/1744] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:39:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:39:24.034170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:24.034919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:24.035155 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:24.045998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] PATH_INFO: `/v3/projects/5e9e91fd716e45938154a7c743d003a5` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:24.050281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e9e91fd716e45938154a7c743d003a5)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:24.052481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:24.113981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:24.114789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:24.115094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:24.172586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:24.173894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:24.191650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e9e91fd716e45938154a7c743d003a5', 'request_id': 'req-745a3a3d-4cdb-4940-9c10-34667024a497'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.211990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e9e91fd716e45938154a7c743d003a5', 'request_id': 'req-745a3a3d-4cdb-4940-9c10-34667024a497'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.237148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e9e91fd716e45938154a7c743d003a5', 'request_id': 'req-745a3a3d-4cdb-4940-9c10-34667024a497'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.261466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] The token cache is being invalidate because project 5e9e91fd716e45938154a7c743d003a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:24.261466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.261466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.270649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 858/1745] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:23 2022] DELETE /identity/v3/projects/5e9e91fd716e45938154a7c743d003a5 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:24.271696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:24.273708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:24.419136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a3a3d-4cdb-4940-9c10-34667024a497 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:24.439603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:24.446282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:24.446582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:24.446860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] PATH_INFO: `/v3/users/3c54656e6d5a46e699ca2653ec8f93c5` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:24.451495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c54656e6d5a46e699ca2653ec8f93c5)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:24.456189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:24.456456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 888/1746] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:39:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 343 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:24.546370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:24.573577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:24.575669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:24.577613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c54656e6d5a46e699ca2653ec8f93c5'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.590034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:24.596509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c54656e6d5a46e699ca2653ec8f93c5'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.604128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e792b22-f00d-43f0-b593-7bc4b2b9dc93 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:24.621397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:24.622545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:24.623013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:24.623455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:24.635797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c54656e6d5a46e699ca2653ec8f93c5'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.638247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c54656e6d5a46e699ca2653ec8f93c5'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.640965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:24.646977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:24.665129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-228bed78-6cdb-4cdb-818a-28ff217f5c75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c54656e6d5a46e699ca2653ec8f93c5'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:24.680428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 859/1747] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:24 2022] DELETE /identity/v3/users/3c54656e6d5a46e699ca2653ec8f93c5 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:24.681964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 889/1748] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:24.896610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5a3106-c11c-4875-a3c1-1c726b4856bc None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:24.897309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5a3106-c11c-4875-a3c1-1c726b4856bc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:24.897620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5a3106-c11c-4875-a3c1-1c726b4856bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:24.956571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-af5a3106-c11c-4875-a3c1-1c726b4856bc None None] MFA Rules not processed for user `befee01f638c4e7681b30c64acad6a6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:24.957834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-af5a3106-c11c-4875-a3c1-1c726b4856bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:25.082338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:25.087851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:25.114508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a72b21-08d4-4786-864a-17517f035373 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:25.147590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 860/1749] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:24 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:25.259318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:25.260417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:25.260706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:25.261179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] PATH_INFO: `/v3/projects/0f44ee8cb38e434983339d605b1121c6` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:25.283331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f44ee8cb38e434983339d605b1121c6)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:25.286581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:25.398655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f44ee8cb38e434983339d605b1121c6', 'request_id': 'req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:25.412270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f44ee8cb38e434983339d605b1121c6', 'request_id': 'req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:25.431189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f44ee8cb38e434983339d605b1121c6', 'request_id': 'req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:25.474833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] The token cache is being invalidate because project 0f44ee8cb38e434983339d605b1121c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:25.476240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:25.477520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2a5d53fa-f5b5-4235-aaaf-0fdb0e171e2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:25.483810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 890/1750] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:25 2022] DELETE /identity/v3/projects/0f44ee8cb38e434983339d605b1121c6 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:25.488596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:25.488891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:25.489169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:25.525641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:25.533708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:25.725603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 861/1751] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:25.727324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:25.729836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:25.729836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:25.778120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:25.782183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:25.795923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 891/1752] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:25.797602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:25.824393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f1c68f-63a4-44cd-9b06-ba31f980e03a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:25.848439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:25.849648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:25.849928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:25.850245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:25.850977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:25.851290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:25.853199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:25.906767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 862/1753] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:25.920972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:25.936631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211fa508-6921-4079-81d8-6ade22cb194d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:25.954678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:25.957096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:25.957400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:25.959173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:25.959891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:25.968147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:26.034287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:26.036244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:26.063117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 892/1754] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:25 2022] POST /identity/v3/projects => generated 367 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:26.064471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:26.077686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f9c43d-d967-4d04-bb52-39fbf901ac6e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:26.312285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:26.326914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:26.346398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:26.347133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.347340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d24e30-a29f-4434-a795-bd66aeb3a795 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:26.348413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.349374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] PATH_INFO: `/v3/projects/4305282983c649c3bbabec098e027ca5` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.351144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4305282983c649c3bbabec098e027ca5)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:26.354416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:26.365181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f958e883-9157-494d-8cad-00017fff9547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:26.366383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f958e883-9157-494d-8cad-00017fff9547 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.393920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f958e883-9157-494d-8cad-00017fff9547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.395092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f958e883-9157-494d-8cad-00017fff9547 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.396919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f958e883-9157-494d-8cad-00017fff9547 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:26.400321 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f958e883-9157-494d-8cad-00017fff9547 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:26.469203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 893/1755] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:26 2022] POST /identity/v3/users => generated 284 bytes in 406 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:26.493032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.493501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.496506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.502646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4305282983c649c3bbabec098e027ca5', 'request_id': 'req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:26.519975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4305282983c649c3bbabec098e027ca5', 'request_id': 'req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:26.532123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4305282983c649c3bbabec098e027ca5', 'request_id': 'req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:26.552861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] MFA Rules not processed for user `bf68e214fd4740be96f2d6c4325f6ee9`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:26.556763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] The token cache is being invalidate because project 4305282983c649c3bbabec098e027ca5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:26.557054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:26.558485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:26.564468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 863/1756] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:26 2022] DELETE /identity/v3/projects/4305282983c649c3bbabec098e027ca5 => generated 0 bytes in 528 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:26.565394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:26.570977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:26.602142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:26.771546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a69a8a8-c153-4c93-91cf-f4395dd5ba80 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:26.793885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:26.794525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.794809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.795111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.795776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:26.797420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:26.824233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 864/1757] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:26 2022] GET /identity/v3/roles => generated 1420 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:26.825129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:26.837537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:26.848590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 894/1758] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 358 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:26.849638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:26.867039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561701b9-3d1e-42bf-a2a3-c8fd98aa4e39 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:26.875016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:26.875681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.875893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.876291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] PATH_INFO: `/v3/users/a7cf6befca02444a84f4843d0c5fbfb1` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.876455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7018fe-613d-41fa-a1bb-c75552c63af4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:26.877360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7cf6befca02444a84f4843d0c5fbfb1)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:26.879360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:26.889996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:26.891302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.891590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.891869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.903276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:26.906370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:26.928001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 895/1759] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:26 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:26.929468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:26.949871 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a9cfa-4f12-4349-a742-8c560ce5cb26 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:26.961990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f30167-f4b7-4493-bba8-90a260a62848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:26.964967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f30167-f4b7-4493-bba8-90a260a62848 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:26.965369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f30167-f4b7-4493-bba8-90a260a62848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:26.965760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f30167-f4b7-4493-bba8-90a260a62848 admin admin] PATH_INFO: `/v3/projects/9f4b4aa9bce8413387d4369cf903f9a8/users/bf68fd8b9a434597aaaa2c034c6a066b/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:26.993528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7cf6befca02444a84f4843d0c5fbfb1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:26.998821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f30167-f4b7-4493-bba8-90a260a62848 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f4b4aa9bce8413387d4369cf903f9a8, user_id=bf68fd8b9a434597aaaa2c034c6a066b, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.001990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f30167-f4b7-4493-bba8-90a260a62848 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.005372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7cf6befca02444a84f4843d0c5fbfb1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.020037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 896/1760] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:26 2022] PUT /identity/v3/projects/9f4b4aa9bce8413387d4369cf903f9a8/users/bf68fd8b9a434597aaaa2c034c6a066b/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:27.022603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7cf6befca02444a84f4843d0c5fbfb1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.023333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7cf6befca02444a84f4843d0c5fbfb1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.028790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:27.031171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.031746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.031746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.035050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7cf6befca02444a84f4843d0c5fbfb1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.047486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 865/1761] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:26 2022] DELETE /identity/v3/users/a7cf6befca02444a84f4843d0c5fbfb1 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:27.068857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] MFA Rules not processed for user `bf68fd8b9a434597aaaa2c034c6a066b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:27.070734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.182152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 897/1762] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:27 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:27.183700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.185315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.303093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b5f5f4-860e-460b-b052-07bdd8dc11f0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.311186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.314674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.315820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.316451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.317302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.321005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.365672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 866/1763] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:27 2022] POST /identity/v3/projects => generated 369 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:27.369422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.385847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879d808-16a1-475e-8ee7-6960d7f31679 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.433084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:27.460983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.461679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.462029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.462373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.463128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.465189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.488798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 898/1764] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:27 2022] POST /identity/v3/users => generated 291 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:27.494318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.499627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.502071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.519241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0961c66d-5505-4125-baaa-8bfee299f8b9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.529159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.530143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.530350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.530757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.531596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.533458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.545728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3c9321-d4b2-412e-85f7-d812b2923e17 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.551201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 867/1765] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:27 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:27.553184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.557535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.558590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.559007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.559418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] PATH_INFO: `/v3/projects/7b909cddd2404bb18989d4c1f1d2a21a` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.560725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b909cddd2404bb18989d4c1f1d2a21a)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.563073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.574201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eafbf29-fe97-4c0f-98af-c9dfff4b311b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.580534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.581440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.581710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.583343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] PATH_INFO: `/v3/projects/3ee1b42c6aa84ccdae97b82e1930a310/users/c75fcde6560147fe908375b3ea689147/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.602038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ee1b42c6aa84ccdae97b82e1930a310, user_id=c75fcde6560147fe908375b3ea689147, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.604744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.628107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 868/1766] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:27 2022] PUT /identity/v3/projects/3ee1b42c6aa84ccdae97b82e1930a310/users/c75fcde6560147fe908375b3ea689147/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:27.628107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.631282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.639677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b909cddd2404bb18989d4c1f1d2a21a', 'request_id': 'req-282f15ed-b384-40f0-b9fc-da4b827bad78'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.653105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b909cddd2404bb18989d4c1f1d2a21a', 'request_id': 'req-282f15ed-b384-40f0-b9fc-da4b827bad78'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.663421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b909cddd2404bb18989d4c1f1d2a21a', 'request_id': 'req-282f15ed-b384-40f0-b9fc-da4b827bad78'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.681720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] The token cache is being invalidate because project 7b909cddd2404bb18989d4c1f1d2a21a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:27.682030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.683070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.689356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 899/1767] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:27 2022] DELETE /identity/v3/projects/7b909cddd2404bb18989d4c1f1d2a21a => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:27.690291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.694416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.713291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e2fce5-2bdd-426b-b58a-de7cd5c0cd76 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.795075 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=1,keystone:SELECT=97,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:27.795439 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f15ed-b384-40f0-b9fc-da4b827bad78 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.803050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fddd4c-3afe-4c1f-a3af-5df9d03e59e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.803677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fddd4c-3afe-4c1f-a3af-5df9d03e59e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.804317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fddd4c-3afe-4c1f-a3af-5df9d03e59e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.804317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fddd4c-3afe-4c1f-a3af-5df9d03e59e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.805861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fddd4c-3afe-4c1f-a3af-5df9d03e59e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.807859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fddd4c-3afe-4c1f-a3af-5df9d03e59e1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.818303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.819255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.819713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.819817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 900/1768] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:27 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:27.820264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] PATH_INFO: `/v3/projects/a5628234021d4e2c9aea69638b52d2e2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.820988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.821322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.821599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.823663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5628234021d4e2c9aea69638b52d2e2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.825942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.843195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:27.844493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.852900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 901/1769] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:27.855044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:27.865931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e348a8-5742-4255-9928-267f3bfc2041 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:27.870661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691561b7-1c52-4d9f-93b9-14d9c51eed0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:27.871356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691561b7-1c52-4d9f-93b9-14d9c51eed0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.871635 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691561b7-1c52-4d9f-93b9-14d9c51eed0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.871911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691561b7-1c52-4d9f-93b9-14d9c51eed0c admin admin] PATH_INFO: `/v3/projects/3ee1b42c6aa84ccdae97b82e1930a310/users/c75fcde6560147fe908375b3ea689147/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.873900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691561b7-1c52-4d9f-93b9-14d9c51eed0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ee1b42c6aa84ccdae97b82e1930a310, user_id=c75fcde6560147fe908375b3ea689147, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:27.876587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691561b7-1c52-4d9f-93b9-14d9c51eed0c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:27.885220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5628234021d4e2c9aea69638b52d2e2', 'request_id': 'req-ddae7718-623a-480a-a298-56f8312aa73b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.898107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5628234021d4e2c9aea69638b52d2e2', 'request_id': 'req-ddae7718-623a-480a-a298-56f8312aa73b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.903626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 902/1770] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:27 2022] PUT /identity/v3/projects/3ee1b42c6aa84ccdae97b82e1930a310/users/c75fcde6560147fe908375b3ea689147/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:27.913089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5628234021d4e2c9aea69638b52d2e2', 'request_id': 'req-ddae7718-623a-480a-a298-56f8312aa73b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.914623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879a6cd-7db1-4087-8d06-780cba3d8fe7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.915162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879a6cd-7db1-4087-8d06-780cba3d8fe7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.915646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879a6cd-7db1-4087-8d06-780cba3d8fe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.926110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] The token cache is being invalidate because project a5628234021d4e2c9aea69638b52d2e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:27.926416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.927660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ddae7718-623a-480a-a298-56f8312aa73b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:27.933523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 869/1771] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:27 2022] DELETE /identity/v3/projects/a5628234021d4e2c9aea69638b52d2e2 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:27.933523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054819ef-bac7-417c-adef-40336a1f59af None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:27.933523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054819ef-bac7-417c-adef-40336a1f59af None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:27.933523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054819ef-bac7-417c-adef-40336a1f59af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:27.937348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2879a6cd-7db1-4087-8d06-780cba3d8fe7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:27.939027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2879a6cd-7db1-4087-8d06-780cba3d8fe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:27.970860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-054819ef-bac7-417c-adef-40336a1f59af None None] MFA Rules not processed for user `c75fcde6560147fe908375b3ea689147`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:27.971684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:27.972553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-054819ef-bac7-417c-adef-40336a1f59af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.034692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 903/1772] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:28.036244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.036631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.037003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.057489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:28.058864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.070857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 904/1773] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:28.071897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.084708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f22839-baa5-4237-ba45-ff85c1184877 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.091087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.091682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.092157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.092445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.093223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:28.093536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.095224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.097827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 870/1774] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:27 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:28.099645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.100242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.101078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.113514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 905/1775] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:28.115203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.117147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.137855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:28.140428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.152005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 871/1776] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:28.155003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.172228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690a0763-2d3c-4cba-b215-1ca340941461 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.177875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.178659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.179267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.179547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.180183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.182111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.205533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bb816c-24cd-423d-9cc9-011154f6476f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.210461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6eaf1a-dd7e-4c6a-afd9-c6ba2b096f19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.211147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6eaf1a-dd7e-4c6a-afd9-c6ba2b096f19 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.211436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6eaf1a-dd7e-4c6a-afd9-c6ba2b096f19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.211936 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6eaf1a-dd7e-4c6a-afd9-c6ba2b096f19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.217417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 872/1777] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/projects => generated 407 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:28.217417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.228052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6eaf1a-dd7e-4c6a-afd9-c6ba2b096f19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.230017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6eaf1a-dd7e-4c6a-afd9-c6ba2b096f19 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.243432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 906/1778] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:28.244546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3eae5e-5928-4b7d-a5e5-563d4c64370e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.244892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3eae5e-5928-4b7d-a5e5-563d4c64370e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.245173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3eae5e-5928-4b7d-a5e5-563d4c64370e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.246420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 907/1779] 10.222.0.20 () {54 vars in 912 bytes} [Fri Sep 9 04:39:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:39:28.251768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3eae5e-5928-4b7d-a5e5-563d4c64370e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.280427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:28.329537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3151a2-0a5d-4af1-b5b3-bb02621a345c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.336941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.337711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.338279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.338559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.339333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:28.339638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.340758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3eae5e-5928-4b7d-a5e5-563d4c64370e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.342195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecae532e-db23-42cf-b42a-926b271979ae admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.346217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.346918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.347338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.347686 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.348397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.351026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.366499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 873/1780] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:28.366499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.366499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.366499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.391164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:28.392498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.395497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 908/1781] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/users => generated 304 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:28.400825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.413631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011694bc-12b7-4f24-bdbc-5755775ed4a8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.420742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.421399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.421835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.422121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.422766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.424745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.466749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 874/1782] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:28.474497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 909/1783] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/projects => generated 383 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:28.474895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.475463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.493467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcedeed2-a384-4920-9341-0ab6c39a04f7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.505696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:28.562809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7fe13a-1dca-4a04-8ff7-bc510f75d335 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.568995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.569838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.570260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.570824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.571749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.573663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.592782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.593568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.594058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.594517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.595372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.597691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.608270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 910/1784] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:28.608270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.608270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.612759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 875/1785] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:28.616278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.629566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9c3cd1-8f73-48f9-a8e0-43991f2d41bb admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.639453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.640070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.640352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.640752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.641421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.643047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.654877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 876/1786] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:28.655711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.671795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d678b5-4b59-4eb9-8b46-7257eccb2257 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.677607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.678223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.678504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.678777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.679462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a74a22-21b1-4558-9976-0de3b267117e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.679543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.681070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.692820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.693376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 877/1787] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:28.694216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.694317 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.694599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.695003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.704293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.705743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.708717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d0d38-2e2b-4209-8c13-7d385d73a9f3 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.714853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.715593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.715938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.716280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.716994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.718032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 911/1788] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:28.718734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.718901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.736078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07708b88-bef0-43f6-8961-94b844f79e3e service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.736260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 878/1789] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:28 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:28.739755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d969f-fbd8-4d2c-9eb7-24ef3937dbcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.740524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:28.740642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d969f-fbd8-4d2c-9eb7-24ef3937dbcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.740834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d969f-fbd8-4d2c-9eb7-24ef3937dbcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.741146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d969f-fbd8-4d2c-9eb7-24ef3937dbcb admin admin] PATH_INFO: `/v3/projects/5165845504364fbca1c38254cb657662/users/982916f8a60a4025a95b5a6c748e548c/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.755206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1108e-7958-4059-872e-f8de18c7bedb admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:28.764043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d969f-fbd8-4d2c-9eb7-24ef3937dbcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=5165845504364fbca1c38254cb657662, user_id=982916f8a60a4025a95b5a6c748e548c, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.767833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93552928-3e70-4a36-9f18-0f29c86c703b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:28.768415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d969f-fbd8-4d2c-9eb7-24ef3937dbcb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.768522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93552928-3e70-4a36-9f18-0f29c86c703b admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.769197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93552928-3e70-4a36-9f18-0f29c86c703b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.769197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93552928-3e70-4a36-9f18-0f29c86c703b admin admin] PATH_INFO: `/v3/projects/d606b4becdbe4b7d8b15a42ef82e7756/users/da199e424e874183bc8bc2462d7622c1/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.788777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93552928-3e70-4a36-9f18-0f29c86c703b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d606b4becdbe4b7d8b15a42ef82e7756, user_id=da199e424e874183bc8bc2462d7622c1, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:28.792455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93552928-3e70-4a36-9f18-0f29c86c703b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:28.799027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 912/1790] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:28 2022] PUT /identity/v3/projects/5165845504364fbca1c38254cb657662/users/982916f8a60a4025a95b5a6c748e548c/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:28.808933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.809222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.809504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.820758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 879/1791] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:28 2022] PUT /identity/v3/projects/d606b4becdbe4b7d8b15a42ef82e7756/users/da199e424e874183bc8bc2462d7622c1/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:28.834185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:28.834478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:28.834756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:28.873507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] MFA Rules not processed for user `da199e424e874183bc8bc2462d7622c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:28.875461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.877942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:28.888606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] MFA Rules not processed for user `982916f8a60a4025a95b5a6c748e548c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:28.891405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:28.984529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 880/1792] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:28.989875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.014851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 913/1793] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:29.018320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.047103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:29.086825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dccaaa6-e586-4c7d-a77f-ed1e9117161f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.094854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.097399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.097399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.097399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.097671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.099556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.116647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6f7acf-05c4-458e-93c4-90855b16d18d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.128504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.129246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.129534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.131043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.132159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.134859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.146023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 881/1794] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:28 2022] POST /identity/v3/projects => generated 381 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:29.155878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.174564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 914/1795] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:29 2022] POST /identity/v3/projects => generated 407 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:29.186193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.277716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9bfa02-998a-4684-bcfc-616d93c7ae06 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.282925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.283663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.283972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.284344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.289007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957128c-f421-4138-ae54-d42d44e82574 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.291941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.293926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.306264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.307907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.308262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.308549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.309797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.312204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.337043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 882/1796] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:29 2022] POST /identity/v3/users => generated 297 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:29.337043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.343330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 915/1797] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:29 2022] POST /identity/v3/users => generated 310 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:29.346282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.349978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ad9d2a-6de9-4067-b2d9-acb4b9a2fe63 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.354824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.355473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.355828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.356019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.356674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.362405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.370522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae899f6-8129-49d9-ac28-ec403d660cf4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.377440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 883/1798] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:29 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:29.379002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.379749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.380032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.380305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.380959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.386987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.396452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 916/1799] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:29 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:29.404110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.408156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.427569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:29.432106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5705b0-90ed-4da1-b6f1-466bb3075b9f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.439275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.440144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.440254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a90d86-81b6-47d9-a3f4-fe6d878b16dd admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.440752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.441099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] PATH_INFO: `/v3/projects/9f996c33780940f882f82cdd8bdfc3d4/users/0f0099efbb8f4d4895107ef89b717f13/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.457065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.458976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.459106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.459474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] PATH_INFO: `/v3/projects/9fafa5ea24bf45139cce26abcda33a7e/users/0d4b151aa6f4437ca877354b04aa93b6/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.479502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f996c33780940f882f82cdd8bdfc3d4, user_id=0f0099efbb8f4d4895107ef89b717f13, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.482547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.500802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fafa5ea24bf45139cce26abcda33a7e, user_id=0d4b151aa6f4437ca877354b04aa93b6, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.502049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 917/1800] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:29 2022] PUT /identity/v3/projects/9f996c33780940f882f82cdd8bdfc3d4/users/0f0099efbb8f4d4895107ef89b717f13/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:29.503642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.506437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.529533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 884/1801] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:29 2022] PUT /identity/v3/projects/9fafa5ea24bf45139cce26abcda33a7e/users/0d4b151aa6f4437ca877354b04aa93b6/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:29.545513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.610174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:29.622455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aebde3-33c2-4fa4-93d4-524d1052775c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.678449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44174a5a-643b-4674-94cb-9bfea5255b40 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.687542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.688248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.688598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.689292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.690021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.691962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.709315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 885/1802] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:29 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:29.713340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.735407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eb3a76-c108-4dc3-a368-574b5bafda2b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.736090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.737269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.737727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.738423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.740716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.743208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.760681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c974f9-3cfc-482b-9073-35eb6776ce44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.760681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c974f9-3cfc-482b-9073-35eb6776ce44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.760681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c974f9-3cfc-482b-9073-35eb6776ce44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.761126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c974f9-3cfc-482b-9073-35eb6776ce44 admin admin] PATH_INFO: `/v3/projects/9fafa5ea24bf45139cce26abcda33a7e/users/0d4b151aa6f4437ca877354b04aa93b6/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.762466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 918/1803] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:29 2022] GET /identity/v3/roles => generated 1420 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:29.764410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.770397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c974f9-3cfc-482b-9073-35eb6776ce44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fafa5ea24bf45139cce26abcda33a7e, user_id=0d4b151aa6f4437ca877354b04aa93b6, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.774856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c974f9-3cfc-482b-9073-35eb6776ce44 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.796493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6db8f-d42d-4b1a-b353-e063e7aedaa4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:29.808855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 886/1804] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:29 2022] PUT /identity/v3/projects/9fafa5ea24bf45139cce26abcda33a7e/users/0d4b151aa6f4437ca877354b04aa93b6/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:29.814714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.815138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.815549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.853094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] MFA Rules not processed for user `0d4b151aa6f4437ca877354b04aa93b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:29.854567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:29.924047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb1d34d-3ea6-4963-b854-acc922374387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:29.925189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb1d34d-3ea6-4963-b854-acc922374387 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.925794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb1d34d-3ea6-4963-b854-acc922374387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.926121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb1d34d-3ea6-4963-b854-acc922374387 admin admin] PATH_INFO: `/v3/projects/9f996c33780940f882f82cdd8bdfc3d4/users/0f0099efbb8f4d4895107ef89b717f13/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.928907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb1d34d-3ea6-4963-b854-acc922374387 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f996c33780940f882f82cdd8bdfc3d4, user_id=0f0099efbb8f4d4895107ef89b717f13, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:29.932648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb1d34d-3ea6-4963-b854-acc922374387 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:29.954965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 919/1805] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:29 2022] PUT /identity/v3/projects/9f996c33780940f882f82cdd8bdfc3d4/users/0f0099efbb8f4d4895107ef89b717f13/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:29.962846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:29.963551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:29.963971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:29.977382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 887/1806] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:29 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:29.990052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:29.991526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:30.053763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] MFA Rules not processed for user `0f0099efbb8f4d4895107ef89b717f13`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:30.055426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:30.137282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:30.147665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9f0b8b-710b-4100-ab2f-d4ba5c2d6038 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:30.198549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 920/1807] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:29 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:30.226469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:30.227130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.227617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.227900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.281451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:30.294715 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e67617b-9a1b-4046-9a5c-44e6dc3bcff9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:30.298918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:30.299593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.299942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.300215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.311791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:30.313381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:30.320756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:30.322188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:30.323565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 921/1808] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:30 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:30.332315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 888/1809] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:29 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:30.462822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:30.464686 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:30.484040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:30.494977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:30.505466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c5bd66-6f90-4b70-a4c5-1ab817a4595e service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:30.510302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:30.510912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.511249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.511477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.552127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e3d954-6ffb-4e4e-8e70-14b5c8303502 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:30.557149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:30.558174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.558456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.558731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] PATH_INFO: `/v3/users/bf68fd8b9a434597aaaa2c034c6a066b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.560254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf68fd8b9a434597aaaa2c034c6a066b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:30.562171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:30.577684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:30.579159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:30.589004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 889/1810] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:30 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:30.589823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:30.597824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f303ee-928b-4c1b-9bcf-604493ae9573 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:30.602099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:30.602860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.603168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.603575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.615078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf68fd8b9a434597aaaa2c034c6a066b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:30.624723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf68fd8b9a434597aaaa2c034c6a066b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:30.642513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf68fd8b9a434597aaaa2c034c6a066b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:30.643295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf68fd8b9a434597aaaa2c034c6a066b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:30.652768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac0bff61-179b-45ff-92bb-9d5820e66aaf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf68fd8b9a434597aaaa2c034c6a066b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:30.659763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 922/1811] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:30 2022] DELETE /identity/v3/users/bf68fd8b9a434597aaaa2c034c6a066b => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:30.663498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:30.664323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.664593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.664945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.674240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:30.675723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:30.684919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 890/1812] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:30 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:30.692712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:30.694002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:30.763556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 923/1813] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:30.774958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:30.776554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:30.839528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e05449-3918-4639-9cc2-d67068201c74 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:30.914204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:30.915121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:30.915490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:30.915808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:30.927450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:30.928946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:30.942867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 891/1814] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:31.361213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:31.363196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:31.379928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98039e1-05e9-401e-8574-d4aa1c02a0d9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:31.399429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:31.400337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:31.400760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:31.401159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] PATH_INFO: `/v3/projects/9f4b4aa9bce8413387d4369cf903f9a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:31.402551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f4b4aa9bce8413387d4369cf903f9a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:31.404907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:31.435187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:31.465998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f4b4aa9bce8413387d4369cf903f9a8', 'request_id': 'req-b67aa408-b713-4694-a7d8-ac0e04d500c4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.473288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f4b4aa9bce8413387d4369cf903f9a8', 'request_id': 'req-b67aa408-b713-4694-a7d8-ac0e04d500c4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.480311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f4b4aa9bce8413387d4369cf903f9a8', 'request_id': 'req-b67aa408-b713-4694-a7d8-ac0e04d500c4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.492749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] The token cache is being invalidate because project 9f4b4aa9bce8413387d4369cf903f9a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:31.493042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.493801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.495361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 924/1815] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:31 2022] DELETE /identity/v3/projects/9f4b4aa9bce8413387d4369cf903f9a8 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:31.500158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:31.509635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68710b5f-bff9-4f11-a0f6-5c2b10058eb9 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:31.567406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:31.568079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:31.568363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:31.568638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] PATH_INFO: `/v3/users/c75fcde6560147fe908375b3ea689147` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:31.569900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c75fcde6560147fe908375b3ea689147)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:31.571895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:31.622016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c75fcde6560147fe908375b3ea689147'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.626617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:31.632748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c75fcde6560147fe908375b3ea689147'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.646628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c75fcde6560147fe908375b3ea689147'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.647720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c75fcde6560147fe908375b3ea689147'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.656503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0a779e8c-5f24-4a9e-92ac-ac259e9c6a09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c75fcde6560147fe908375b3ea689147'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:31.664675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 892/1816] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:31 2022] DELETE /identity/v3/users/c75fcde6560147fe908375b3ea689147 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:31.926537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:31.928104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:31.944047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67aa408-b713-4694-a7d8-ac0e04d500c4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.005263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.005263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.005263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.005263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] PATH_INFO: `/v3/projects/3ee1b42c6aa84ccdae97b82e1930a310` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.005698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ee1b42c6aa84ccdae97b82e1930a310)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.007639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.060953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ee1b42c6aa84ccdae97b82e1930a310', 'request_id': 'req-9ad4d97b-3290-430b-b18a-96e471996a88'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:32.070103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ee1b42c6aa84ccdae97b82e1930a310', 'request_id': 'req-9ad4d97b-3290-430b-b18a-96e471996a88'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:32.079201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ee1b42c6aa84ccdae97b82e1930a310', 'request_id': 'req-9ad4d97b-3290-430b-b18a-96e471996a88'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:32.093031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] The token cache is being invalidate because project 3ee1b42c6aa84ccdae97b82e1930a310 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:32.093323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:32.094014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ad4d97b-3290-430b-b18a-96e471996a88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:32.095536 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 925/1817] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:31 2022] DELETE /identity/v3/projects/3ee1b42c6aa84ccdae97b82e1930a310 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:32.100373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.100692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.100972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.121090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:32.122306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:32.181214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 893/1818] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:32.190408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.190698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.191015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.211644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:32.212839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:32.220600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 926/1819] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:32.224700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.234295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9bd946-53d0-4e08-9127-a11e929e99c3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.238871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.239508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.239792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.240065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.240744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:32.241051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.242649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.254243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 894/1820] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:32.259350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.268928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3317ef6e-fec2-46df-a171-b574061c343e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.274132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.274726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.275033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.275303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.275970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.277843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.303777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 927/1821] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:32.307867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.319652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f0a7cb-cc0b-48c1-abe5-8d6f9a56f8c7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.376650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.377289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.377574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.377847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.378477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.381016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.402662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 895/1822] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:32.406845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.417124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c819ed91-7e52-45a1-80c5-3df8e1714cec admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.421679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.422281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.423275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.423576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.424310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.425900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.435898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:32.442230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 928/1823] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:32.447254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.457810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e069af9-5163-49cf-9b40-5b4f17357232 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.462497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c445d7c2-5124-42fd-8f3b-e6692dee2fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.463142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c445d7c2-5124-42fd-8f3b-e6692dee2fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.463430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c445d7c2-5124-42fd-8f3b-e6692dee2fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.463704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c445d7c2-5124-42fd-8f3b-e6692dee2fa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.464350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c445d7c2-5124-42fd-8f3b-e6692dee2fa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.465955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c445d7c2-5124-42fd-8f3b-e6692dee2fa7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.481785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.487123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 896/1824] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:32 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:32.492890 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331f46c8-5a59-4e70-b824-b359f36684d1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.507441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.508128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.508412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.508692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] PATH_INFO: `/v3/projects/61a5e92c86694acf97d2489b793d57a3/users/d0a203d93b1e4e3db7a499663ae05fe8/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.526417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=61a5e92c86694acf97d2489b793d57a3, user_id=d0a203d93b1e4e3db7a499663ae05fe8, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.529091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.544658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 929/1825] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:32 2022] PUT /identity/v3/projects/61a5e92c86694acf97d2489b793d57a3/users/d0a203d93b1e4e3db7a499663ae05fe8/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:32.555360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.555863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.555941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.586321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] MFA Rules not processed for user `d0a203d93b1e4e3db7a499663ae05fe8`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:32.587577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:32.612984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:32.677674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 897/1826] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:32.688335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.784786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bd3120-46dd-4d5e-a916-dba56da5f1bb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.789570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.790188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.790488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.790755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.791438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.793385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:32.834416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:32.835668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 930/1827] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/projects => generated 385 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:32.855547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603e694a-ef30-4693-a7d0-f89cdc1ca4e7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:32.962083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:32.962774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:32.963160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:32.963354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:32.964005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:32.971662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.000286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 898/1828] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:32 2022] POST /identity/v3/users => generated 299 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:33.004138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.016329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7927f-f45c-4ce9-84c6-ebc1a18e29ef admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.031594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.032234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.032681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.032962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.033961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.035555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.054419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 931/1829] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:33 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:33.055557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.070646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db32f1-12ba-4bac-9a7d-53ba5020a619 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.076149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.077652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.077652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.077652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] PATH_INFO: `/v3/projects/49002939f01c4ca09a0e5fcc9c62c304/users/9dcb0d4f55c049f0acaf0c5e1fbdf89b/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.096489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] RBAC: Authorizing `identity:create_grant(project_id=49002939f01c4ca09a0e5fcc9c62c304, user_id=9dcb0d4f55c049f0acaf0c5e1fbdf89b, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.099414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.113717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 899/1830] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:33 2022] PUT /identity/v3/projects/49002939f01c4ca09a0e5fcc9c62c304/users/9dcb0d4f55c049f0acaf0c5e1fbdf89b/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:33.118332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.128585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f369584-1b72-4430-a0b6-33204275fc57 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.207429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d348fd1f-08f9-4197-a405-f92d54bdf971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.209492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d348fd1f-08f9-4197-a405-f92d54bdf971 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.209780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d348fd1f-08f9-4197-a405-f92d54bdf971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.210186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d348fd1f-08f9-4197-a405-f92d54bdf971 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.212808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d348fd1f-08f9-4197-a405-f92d54bdf971 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.216985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d348fd1f-08f9-4197-a405-f92d54bdf971 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.233595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 932/1831] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:33 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:33.237941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.255989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2302ae4-d640-4092-9db2-a68660fd037d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.263254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.265394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.265894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.266356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] PATH_INFO: `/v3/projects/49002939f01c4ca09a0e5fcc9c62c304/users/9dcb0d4f55c049f0acaf0c5e1fbdf89b/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.269096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49002939f01c4ca09a0e5fcc9c62c304, user_id=9dcb0d4f55c049f0acaf0c5e1fbdf89b, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.272017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.286969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 900/1832] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:33 2022] PUT /identity/v3/projects/49002939f01c4ca09a0e5fcc9c62c304/users/9dcb0d4f55c049f0acaf0c5e1fbdf89b/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:33.295706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.296065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.296357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.330971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] MFA Rules not processed for user `9dcb0d4f55c049f0acaf0c5e1fbdf89b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:33.332260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:33.377340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:33.421794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 933/1833] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:33.433217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.434985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:33.492670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea60fc1-d1d3-469c-96ae-73240b13ac21 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.496414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa1fcba-aeee-4697-b4bc-345676e9f672 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.496949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa1fcba-aeee-4697-b4bc-345676e9f672 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.497230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa1fcba-aeee-4697-b4bc-345676e9f672 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.497486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa1fcba-aeee-4697-b4bc-345676e9f672 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.507247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa1fcba-aeee-4697-b4bc-345676e9f672 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.509302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa1fcba-aeee-4697-b4bc-345676e9f672 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.518724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 901/1834] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:33 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:33.588375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.663776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0d308a-65bf-4482-8a66-df39c4a031e0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.668906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.669603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.669883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.670157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] PATH_INFO: `/v3/users/d0a203d93b1e4e3db7a499663ae05fe8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.671372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0a203d93b1e4e3db7a499663ae05fe8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.673349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.730261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0a203d93b1e4e3db7a499663ae05fe8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:33.741014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0a203d93b1e4e3db7a499663ae05fe8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:33.755763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0a203d93b1e4e3db7a499663ae05fe8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:33.756543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0a203d93b1e4e3db7a499663ae05fe8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:33.772097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0a203d93b1e4e3db7a499663ae05fe8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:33.780726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 934/1835] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:33 2022] DELETE /identity/v3/users/d0a203d93b1e4e3db7a499663ae05fe8 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:33.785077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.785553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.785775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.813385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:33.814999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:33.830355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:33.882673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 902/1836] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:33.892256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:33.893957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:33.953584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feab8096-7513-4c59-98ed-f0a5ab10e4ee admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:33.957361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:33.958011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:33.958292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:33.958567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:33.965000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:33.970127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:33.972617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:33.982707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 935/1837] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:34.377341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:34.378804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:34.398834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c20b5d3-3a96-40b3-9dc3-6543707028d4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:34.416806 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:34.418116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:34.418409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:34.418737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] PATH_INFO: `/v3/projects/61a5e92c86694acf97d2489b793d57a3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:34.421055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] RBAC: Authorizing `identity:delete_project(project_id=61a5e92c86694acf97d2489b793d57a3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:34.424896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:34.527300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61a5e92c86694acf97d2489b793d57a3', 'request_id': 'req-6dec4285-407b-45e2-a5dd-7452365d480c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.546318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61a5e92c86694acf97d2489b793d57a3', 'request_id': 'req-6dec4285-407b-45e2-a5dd-7452365d480c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.559668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61a5e92c86694acf97d2489b793d57a3', 'request_id': 'req-6dec4285-407b-45e2-a5dd-7452365d480c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.580690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] The token cache is being invalidate because project 61a5e92c86694acf97d2489b793d57a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:34.581121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.582034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.584836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 903/1838] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:34 2022] DELETE /identity/v3/projects/61a5e92c86694acf97d2489b793d57a3 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:34.592475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:34.606228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef253ae-e48d-488f-b131-3c49363101b0 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:34.681136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:34.682188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:34.682463 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:34.682737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] PATH_INFO: `/v3/users/9dcb0d4f55c049f0acaf0c5e1fbdf89b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:34.683994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9dcb0d4f55c049f0acaf0c5e1fbdf89b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:34.685932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:34.740011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9dcb0d4f55c049f0acaf0c5e1fbdf89b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.747287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9dcb0d4f55c049f0acaf0c5e1fbdf89b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.760228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9dcb0d4f55c049f0acaf0c5e1fbdf89b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.760841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9dcb0d4f55c049f0acaf0c5e1fbdf89b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.767834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7fb3369c-b493-4ffb-818b-cb5dc8b41343 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9dcb0d4f55c049f0acaf0c5e1fbdf89b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:34.773709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 936/1839] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:34 2022] DELETE /identity/v3/users/9dcb0d4f55c049f0acaf0c5e1fbdf89b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:35.309162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.310785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.324699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dec4285-407b-45e2-a5dd-7452365d480c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.382821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.383435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.383712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.383981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] PATH_INFO: `/v3/projects/49002939f01c4ca09a0e5fcc9c62c304` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.385085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] RBAC: Authorizing `identity:delete_project(project_id=49002939f01c4ca09a0e5fcc9c62c304)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.387005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.429614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:35.433352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49002939f01c4ca09a0e5fcc9c62c304', 'request_id': 'req-e7edf7dd-999b-44c7-8276-9174d75b1391'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.441703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49002939f01c4ca09a0e5fcc9c62c304', 'request_id': 'req-e7edf7dd-999b-44c7-8276-9174d75b1391'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.447394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49002939f01c4ca09a0e5fcc9c62c304', 'request_id': 'req-e7edf7dd-999b-44c7-8276-9174d75b1391'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.458830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] The token cache is being invalidate because project 49002939f01c4ca09a0e5fcc9c62c304 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:35.459184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.459844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e7edf7dd-999b-44c7-8276-9174d75b1391 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.461350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 904/1840] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:35 2022] DELETE /identity/v3/projects/49002939f01c4ca09a0e5fcc9c62c304 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:35.466230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.466521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.466809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.486662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:35.488008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.546533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 937/1841] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:35.554008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.554305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.554556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.574498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:35.575683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.584939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 905/1842] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:35.589040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.600042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90471cf6-85b5-4184-b15e-ff5485248b0c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.605309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.605992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.606325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.606656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.607394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:35.607771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.609416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.623792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:35.636534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 938/1843] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:35.637407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.649764 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0cd982-242d-4e28-b39b-4add34d3a524 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.654816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.655061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.655497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.655787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.656211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.656811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.656915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.658754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.673632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5903731-016b-4221-bb51-37528b933da9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.678843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.679518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.679812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.680144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] PATH_INFO: `/v3/users/da199e424e874183bc8bc2462d7622c1` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.681390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da199e424e874183bc8bc2462d7622c1)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.683433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.690066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 906/1844] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:35 2022] POST /identity/v3/projects => generated 381 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:35.694658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.734429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da199e424e874183bc8bc2462d7622c1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.742323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da199e424e874183bc8bc2462d7622c1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.756187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da199e424e874183bc8bc2462d7622c1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.756860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da199e424e874183bc8bc2462d7622c1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.757033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91565937-066c-48ac-a41f-b19f79426652 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.761812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.762417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.762696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.762996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.763633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.763840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ad6b84ad-e1d2-4efa-8a24-360947d76859 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da199e424e874183bc8bc2462d7622c1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:35.765548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.769739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 939/1845] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:35 2022] DELETE /identity/v3/users/da199e424e874183bc8bc2462d7622c1 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:35.774859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.775258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.775534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.791126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 907/1846] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:35 2022] POST /identity/v3/users => generated 291 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:35.792143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.793760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.797986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:35.799223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.805367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d50c2b5-16f0-4ff4-841a-dce234e33a01 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.859951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 940/1847] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:35.864778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.865515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.865805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.866078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.866755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.868413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.870228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.871722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:35.878626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 908/1848] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:35.882503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.892011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13f9711-7b57-4323-ae03-0d3d2e0d2010 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.896597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.897184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.897465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.897739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.898372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.899991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.909598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 909/1849] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:35.914106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:35.923825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6c511f-9b9b-42ed-846b-e4339643f0ef admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.929003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.929367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333fa52-d502-42ca-b998-8cee9dcfa31c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:35.929626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.929982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.930188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] PATH_INFO: `/v3/projects/bad0f8c02d984ef88b9393269d9ec903/users/5687eb6cfd2f4620ac6683fbd9731637/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.933012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8590ac0c-fb0e-4091-93ac-90a0dcf52ebc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:35.933613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8590ac0c-fb0e-4091-93ac-90a0dcf52ebc service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.933894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8590ac0c-fb0e-4091-93ac-90a0dcf52ebc service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.934170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8590ac0c-fb0e-4091-93ac-90a0dcf52ebc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:35.943619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8590ac0c-fb0e-4091-93ac-90a0dcf52ebc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.945043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8590ac0c-fb0e-4091-93ac-90a0dcf52ebc service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.945968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bad0f8c02d984ef88b9393269d9ec903, user_id=5687eb6cfd2f4620ac6683fbd9731637, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:35.948635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:35.951794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=4,keystone:SELECT=90,keystone:INSERT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:35.956016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 941/1850] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:35.964999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 910/1851] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:35 2022] PUT /identity/v3/projects/bad0f8c02d984ef88b9393269d9ec903/users/5687eb6cfd2f4620ac6683fbd9731637/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:35.975894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:35.976269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:35.976452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.016770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] MFA Rules not processed for user `5687eb6cfd2f4620ac6683fbd9731637`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:36.017999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:36.022769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:36.107515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 942/1852] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:35 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:36.108507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.195425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b981a05-3770-4133-bc15-975279bc3836 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.200143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.200748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.201030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.201305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.201952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.203918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.233204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 911/1853] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:36 2022] POST /identity/v3/projects => generated 381 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:36.237036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.250902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27357290-4fa9-4e61-85d8-579db90f1eb6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.313191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.314143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.314484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.314831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.315623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.317805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.343792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 943/1854] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:36 2022] POST /identity/v3/users => generated 297 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:36.344967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.357326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1b6db-920f-44c8-b792-bd07d137bd0a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.362621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.363526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.363987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.364267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.364940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.366798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.378228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 912/1855] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:36.382485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.393535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b699aa-2f8a-41c4-9dc3-de43a36a7329 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.398771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.399462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.399740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.400042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] PATH_INFO: `/v3/projects/79925e3143f64e738beee34ccad9db1e/users/2efd1e1458894bde88467b0a99c025f7/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.415741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79925e3143f64e738beee34ccad9db1e, user_id=2efd1e1458894bde88467b0a99c025f7, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.415875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.417711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:36.418398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.432464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 944/1856] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:36 2022] PUT /identity/v3/projects/79925e3143f64e738beee34ccad9db1e/users/2efd1e1458894bde88467b0a99c025f7/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:36.435681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2d4363-e32f-4494-a93a-e51c602a8405 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.436646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.523921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.524676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.524959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.525234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] PATH_INFO: `/v3/projects/d606b4becdbe4b7d8b15a42ef82e7756` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.525324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2777e753-8364-4ca6-9bef-12d649466af2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.526431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d606b4becdbe4b7d8b15a42ef82e7756)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.528586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.529992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.530598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.530885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.531200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.531851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.533431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.543977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 945/1857] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:36 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:36.547928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.558590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9803acab-1f9d-4c0c-b59d-76e123e22279 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.563174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd162fe-ea01-4ef9-a534-12c523c01a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.564060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd162fe-ea01-4ef9-a534-12c523c01a24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.564339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd162fe-ea01-4ef9-a534-12c523c01a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.564605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd162fe-ea01-4ef9-a534-12c523c01a24 admin admin] PATH_INFO: `/v3/projects/79925e3143f64e738beee34ccad9db1e/users/2efd1e1458894bde88467b0a99c025f7/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.566498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd162fe-ea01-4ef9-a534-12c523c01a24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79925e3143f64e738beee34ccad9db1e, user_id=2efd1e1458894bde88467b0a99c025f7, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.569278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd162fe-ea01-4ef9-a534-12c523c01a24 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.571546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:36.577762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd606b4becdbe4b7d8b15a42ef82e7756', 'request_id': 'req-3a7151a4-3925-4f34-b530-35963542d25f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.581582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 946/1858] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:36 2022] PUT /identity/v3/projects/79925e3143f64e738beee34ccad9db1e/users/2efd1e1458894bde88467b0a99c025f7/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:36.587560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd606b4becdbe4b7d8b15a42ef82e7756', 'request_id': 'req-3a7151a4-3925-4f34-b530-35963542d25f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.595617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd606b4becdbe4b7d8b15a42ef82e7756', 'request_id': 'req-3a7151a4-3925-4f34-b530-35963542d25f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.595813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.596231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.596786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.606767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:36.612263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] The token cache is being invalidate because project d606b4becdbe4b7d8b15a42ef82e7756 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:36.612975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.613916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.616712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 913/1859] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:36 2022] DELETE /identity/v3/projects/d606b4becdbe4b7d8b15a42ef82e7756 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:36.620743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.631222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:36.643229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] MFA Rules not processed for user `2efd1e1458894bde88467b0a99c025f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:36.644445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:36.712269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7151a4-3925-4f34-b530-35963542d25f admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.720805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 947/1860] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:36 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:36.734287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:36.735752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:36.779015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.779745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.780094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.780437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] PATH_INFO: `/v3/users/0d4b151aa6f4437ca877354b04aa93b6` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.781657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d4b151aa6f4437ca877354b04aa93b6)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.783636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.808178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd65cb-6967-4695-9006-7b83e24507d1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:36.812246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:36.812860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:36.813129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:36.813556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:36.824446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:36.825878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:36.836113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 948/1861] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:36 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:36.840510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d4b151aa6f4437ca877354b04aa93b6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.851559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d4b151aa6f4437ca877354b04aa93b6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.865944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d4b151aa6f4437ca877354b04aa93b6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.866652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d4b151aa6f4437ca877354b04aa93b6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.875247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d4b151aa6f4437ca877354b04aa93b6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:36.881967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 914/1862] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:36 2022] DELETE /identity/v3/users/0d4b151aa6f4437ca877354b04aa93b6 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:37.209266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.219910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518cf7ab-38a1-4f66-bb93-46c86615f798 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.228207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.229742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:37.267176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:37.303796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2efd1e1458894bde88467b0a99c025f7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79925e3143f64e738beee34ccad9db1e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.304409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.304692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.306019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.306019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:37.306019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.307488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.309349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37739f8-c624-47ee-9cda-1f9316796039 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.315074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.315391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.315671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.315944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] PATH_INFO: `/v3/projects/9fafa5ea24bf45139cce26abcda33a7e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.317885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fafa5ea24bf45139cce26abcda33a7e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.319708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.321627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 949/1863] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:37.325636 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.329694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:37.343820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577db7b1-32ba-4fd6-a20a-a22e63128155 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.349874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2efd1e1458894bde88467b0a99c025f7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79925e3143f64e738beee34ccad9db1e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.350513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.350785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.351117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.351763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.353649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.369848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:37.376526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fafa5ea24bf45139cce26abcda33a7e', 'request_id': 'req-86a9df9c-72e6-49e5-ac83-a414dd8825dc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.384481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fafa5ea24bf45139cce26abcda33a7e', 'request_id': 'req-86a9df9c-72e6-49e5-ac83-a414dd8825dc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.390985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 950/1864] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:37 2022] POST /identity/v3/projects => generated 353 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:37.391245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fafa5ea24bf45139cce26abcda33a7e', 'request_id': 'req-86a9df9c-72e6-49e5-ac83-a414dd8825dc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.392702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.402719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] The token cache is being invalidate because project 9fafa5ea24bf45139cce26abcda33a7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:37.403109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.403813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-86a9df9c-72e6-49e5-ac83-a414dd8825dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.405603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 915/1865] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:37 2022] DELETE /identity/v3/projects/9fafa5ea24bf45139cce26abcda33a7e => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:37.409835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.410183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.410523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.429182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:37.430563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:37.458398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd2e00a-9787-417e-8eec-09d9cb79863e tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.463009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2efd1e1458894bde88467b0a99c025f7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79925e3143f64e738beee34ccad9db1e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.463619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.463903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.464183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.464825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:37.465131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.466728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb3a390-7b50-40f4-92ce-e51fd7826474 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.483167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 951/1866] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:39:37.511162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 916/1867] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:37.524393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.524393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.524393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.525234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.530712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:37.547520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:37.550278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:37.554289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f72206f-d1ff-43d9-8764-597b98d7f4c6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.561915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2efd1e1458894bde88467b0a99c025f7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79925e3143f64e738beee34ccad9db1e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.562534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 952/1868] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:37.562665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.563172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.563511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] PATH_INFO: `/v3/projects/3bfad7e8e46549f2aea6c706431f85c5` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.565047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorizing `identity:delete_project(project_id=3bfad7e8e46549f2aea6c706431f85c5)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.567587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.568609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.579578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5651356e-ce4a-47b0-b4d4-ab7993403f16 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.585302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.585928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.586268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.586612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.587410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:37.588036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.589723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.606422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 953/1869] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:39:37.607584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.617694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb92905-7da8-47fe-89c2-bbf974dad657 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.635971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bfad7e8e46549f2aea6c706431f85c5', 'request_id': 'req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.642375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:37.675864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bfad7e8e46549f2aea6c706431f85c5', 'request_id': 'req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.696153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bfad7e8e46549f2aea6c706431f85c5', 'request_id': 'req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.729289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] The token cache is being invalidate because project 3bfad7e8e46549f2aea6c706431f85c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:37.729713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.730838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:37.732192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.732980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 917/1870] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:37 2022] DELETE /identity/v3/projects/3bfad7e8e46549f2aea6c706431f85c5 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:37.733605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.734061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.734462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.735273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.737422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:37.783892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 954/1871] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:37 2022] POST /identity/v3/projects => generated 373 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:37.788632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:37.790621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:37.817116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf1c53b-d4a0-4149-8daa-0de1e8c401f2 tempest-VolumeQuotaClassesTest-1884864369 tempest-VolumeQuotaClassesTest-1884864369-project-admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:37.963305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:37.963947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:37.964870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:37.965539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:37.966220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:37.970003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.010084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 918/1872] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:37 2022] POST /identity/v3/users => generated 287 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:38.011453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:38.028827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ec068f-ce01-4be1-850d-147fbbccb7b0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:38.046634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:38.047435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.047778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.049612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.051015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:38.053194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.084380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:38.084956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 955/1873] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:38 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:38.099828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ea67a-b6a9-4b40-9ca9-c04d8464e0df admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:38.116630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:38.117975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.119415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.119875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.125790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:38.127791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.144582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 919/1874] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:38 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:38.148652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:38.161405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692e9844-db66-4a34-a46e-f69ce9a2d53c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:38.171388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3cebd0-cd1d-4e10-9b8d-4309b4a0e776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:38.172106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3cebd0-cd1d-4e10-9b8d-4309b4a0e776 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.172407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:38.172555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3cebd0-cd1d-4e10-9b8d-4309b4a0e776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.173022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3cebd0-cd1d-4e10-9b8d-4309b4a0e776 admin admin] PATH_INFO: `/v3/projects/79ddbc4671b44eeba54e2eeca5387e5c/users/abd58337f6804b97b1a665647b6dce3f/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.175289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:38.193508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3cebd0-cd1d-4e10-9b8d-4309b4a0e776 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79ddbc4671b44eeba54e2eeca5387e5c, user_id=abd58337f6804b97b1a665647b6dce3f, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:38.197344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3cebd0-cd1d-4e10-9b8d-4309b4a0e776 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.234585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c833af8-b0bf-4f0b-b973-c314f8872721 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:38.246277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 956/1875] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:38 2022] PUT /identity/v3/projects/79ddbc4671b44eeba54e2eeca5387e5c/users/abd58337f6804b97b1a665647b6dce3f/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:38.248571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.248925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.249114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.302065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:38.303344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:38.318685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:38.361389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:38.362104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.362477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.363331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] PATH_INFO: `/v3/users/5687eb6cfd2f4620ac6683fbd9731637` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.366090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5687eb6cfd2f4620ac6683fbd9731637)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:38.368079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.397533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 957/1876] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:38.408619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:38.410421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:38.425551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5687eb6cfd2f4620ac6683fbd9731637'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:38.436079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5687eb6cfd2f4620ac6683fbd9731637'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:38.443638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=12,keystone:SELECT=84,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:38.452350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5687eb6cfd2f4620ac6683fbd9731637'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:38.453230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5687eb6cfd2f4620ac6683fbd9731637'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:38.465076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9913b449-edbe-4c92-8658-3b123e02a168 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5687eb6cfd2f4620ac6683fbd9731637'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:38.476908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 920/1877] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:38 2022] DELETE /identity/v3/users/5687eb6cfd2f4620ac6683fbd9731637 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:38.481229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398dbf4-37e4-4b1a-95a5-93f2a3b93524 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:38.487188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218514eb-4883-456e-8833-5b3155d65aff None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.487416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218514eb-4883-456e-8833-5b3155d65aff None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.487688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218514eb-4883-456e-8833-5b3155d65aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.507271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-218514eb-4883-456e-8833-5b3155d65aff None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:38.508463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-218514eb-4883-456e-8833-5b3155d65aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:38.531123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:38.531732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.532013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.532289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.534060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:38.579249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 921/1878] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:38.589271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218514eb-4883-456e-8833-5b3155d65aff None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:38.592271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-218514eb-4883-456e-8833-5b3155d65aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:38.611909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218514eb-4883-456e-8833-5b3155d65aff None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:38.616323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:38.617745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:38.617745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:38.618307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:38.624126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:38.625613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.638046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:38.641780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:38.659197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 958/1879] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:38.661749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 922/1880] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:39.670131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:39.672968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:39.698473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5330d63c-14e4-40c8-ace9-d3fbdd993fcd service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:39.718519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:39.721633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:39.722255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:39.722797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] PATH_INFO: `/v3/projects/bad0f8c02d984ef88b9393269d9ec903` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:39.734899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=bad0f8c02d984ef88b9393269d9ec903)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:39.737433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:39.830469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bad0f8c02d984ef88b9393269d9ec903', 'request_id': 'req-60e1724d-6a5a-46db-92bb-876e8fb87a2b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:39.849631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bad0f8c02d984ef88b9393269d9ec903', 'request_id': 'req-60e1724d-6a5a-46db-92bb-876e8fb87a2b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:39.861884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bad0f8c02d984ef88b9393269d9ec903', 'request_id': 'req-60e1724d-6a5a-46db-92bb-876e8fb87a2b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:39.876868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] The token cache is being invalidate because project bad0f8c02d984ef88b9393269d9ec903 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:39.877308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:39.878040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-60e1724d-6a5a-46db-92bb-876e8fb87a2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:39.879808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 923/1881] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:39 2022] DELETE /identity/v3/projects/bad0f8c02d984ef88b9393269d9ec903 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:39.884490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:39.903880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be699b78-18d2-4986-9901-79878c42bc45 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:39.940933 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:39.999416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:40.000879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:40.001175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:40.001598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] PATH_INFO: `/v3/users/2efd1e1458894bde88467b0a99c025f7` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:40.003007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2efd1e1458894bde88467b0a99c025f7)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:40.006718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:40.164868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2efd1e1458894bde88467b0a99c025f7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:40.178591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2efd1e1458894bde88467b0a99c025f7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:40.198603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2efd1e1458894bde88467b0a99c025f7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:40.199327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2efd1e1458894bde88467b0a99c025f7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:40.208340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79437c35-fafa-42f2-9e6d-47a7d7a5d09d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2efd1e1458894bde88467b0a99c025f7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:40.215939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 959/1882] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:39 2022] DELETE /identity/v3/users/2efd1e1458894bde88467b0a99c025f7 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:40.571443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d7446-fdf3-493c-b021-a16464ab1911 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:40.571959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d7446-fdf3-493c-b021-a16464ab1911 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:40.572389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d7446-fdf3-493c-b021-a16464ab1911 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:40.574181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 924/1883] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:39:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:39:40.588578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ea4d0-084b-477a-8628-d5e814398c77 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:40.588872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ea4d0-084b-477a-8628-d5e814398c77 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:40.589154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ea4d0-084b-477a-8628-d5e814398c77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:40.614759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c33ea4d0-084b-477a-8628-d5e814398c77 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:40.616062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c33ea4d0-084b-477a-8628-d5e814398c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:40.692904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d7446-fdf3-493c-b021-a16464ab1911 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:40.695445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6d2d7446-fdf3-493c-b021-a16464ab1911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:40.738119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 960/1884] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:39:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:40.813614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:40.893405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d7446-fdf3-493c-b021-a16464ab1911 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:40.900643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:40.901299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:40.901640 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:40.902007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] PATH_INFO: `/v3/projects/79925e3143f64e738beee34ccad9db1e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:40.903649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79925e3143f64e738beee34ccad9db1e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:40.906716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:40.963061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:40.963061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:40.963712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.001854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79925e3143f64e738beee34ccad9db1e', 'request_id': 'req-b5dc07f4-b431-4bc0-b432-ab63613473d2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:41.017869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79925e3143f64e738beee34ccad9db1e', 'request_id': 'req-b5dc07f4-b431-4bc0-b432-ab63613473d2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:41.031707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] MFA Rules not processed for user `abd58337f6804b97b1a665647b6dce3f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:41.033025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79925e3143f64e738beee34ccad9db1e', 'request_id': 'req-b5dc07f4-b431-4bc0-b432-ab63613473d2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:41.033243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.066023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] The token cache is being invalidate because project 79925e3143f64e738beee34ccad9db1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:41.066780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:41.068081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b5dc07f4-b431-4bc0-b432-ab63613473d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:41.077906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 925/1885] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:40 2022] DELETE /identity/v3/projects/79925e3143f64e738beee34ccad9db1e => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:41.081462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e023492d-1a46-43c4-ba01-a207e5a8e205 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.081750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e023492d-1a46-43c4-ba01-a207e5a8e205 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.082029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e023492d-1a46-43c4-ba01-a207e5a8e205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.120404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:41.148044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e023492d-1a46-43c4-ba01-a207e5a8e205 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:41.151664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e023492d-1a46-43c4-ba01-a207e5a8e205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.238180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 961/1886] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:41.242039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.244605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.315057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 926/1887] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:41.318123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.318414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.318696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:41.373157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.402829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 927/1888] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:41.405449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.423460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ee08be-fa6f-416e-90c4-9f9ee0c0da28 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.439563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.440297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.440849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.441226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.442603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:41.443034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.446667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.460829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b293d-a92f-4672-89df-3ceb85844e4d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.473984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.475856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.476199 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.476526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.480963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 928/1889] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:41.481395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.481898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.482547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.495348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6356061-688c-4a76-ba67-7d4868a16529 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.503759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.504493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.505261 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.505735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.506397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.509170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.560079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 962/1890] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:41 2022] POST /identity/v3/projects => generated 373 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:41.561659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.564414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.594060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 929/1891] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:41 2022] POST /identity/v3/projects => generated 371 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:41.595972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.679094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:41.736045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db620f2-44a3-448c-91a2-6e66346962c8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.741359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974994fe-c083-4af9-9a52-4a0406373ab3 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.750181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.751280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.752919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.753209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.754996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.757738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.791856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 930/1892] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:41 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:41.792678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.814349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:41.814590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497d4ac7-5639-4811-91ad-c19a31251b54 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.826739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.827486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.827888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.828251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.829227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.832187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.840807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.841441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.841721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.841990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.844095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.858233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.860011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.871343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 931/1893] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:41 2022] POST /identity/v3/users => generated 286 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:41.874386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.884789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 963/1894] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:41.885770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.896522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9f3c78-4745-4cf9-bafc-9d4fccb55cb7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.909529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6886a75e-3dbd-4905-aaeb-a16e9daa200f service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.911568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.912597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.912889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.913174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.913941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.916079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.923262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.924141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.924566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.926093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:41.927651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:41.934219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:41.936293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 932/1895] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:41 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:41.937553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.948273 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 964/1896] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:41 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:41.950466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:41.953566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:41.979939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354dbdc-71f8-4953-a03f-c9eeeced824a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.988096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a768ca4-a0a7-4eae-afe2-550fbc75a057 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:41.992458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:41.993181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:41.993432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:41.993711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] PATH_INFO: `/v3/projects/534865bd729542edb652f63af856077e/users/b9519a147a004ba3a30b0c5eaa65aff4/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.002667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.003494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.003774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.004046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] PATH_INFO: `/v3/users/982916f8a60a4025a95b5a6c748e548c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.005474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] RBAC: Authorizing `identity:delete_user(user_id=982916f8a60a4025a95b5a6c748e548c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:42.007455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:42.028312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=534865bd729542edb652f63af856077e, user_id=b9519a147a004ba3a30b0c5eaa65aff4, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:42.031323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:42.072380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 965/1897] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:41 2022] PUT /identity/v3/projects/534865bd729542edb652f63af856077e/users/b9519a147a004ba3a30b0c5eaa65aff4/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:42.073159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:42.106569 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '982916f8a60a4025a95b5a6c748e548c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:42.120954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '982916f8a60a4025a95b5a6c748e548c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:42.142438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '982916f8a60a4025a95b5a6c748e548c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:42.143760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '982916f8a60a4025a95b5a6c748e548c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:42.155636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '982916f8a60a4025a95b5a6c748e548c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:42.167800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 933/1898] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:41 2022] DELETE /identity/v3/users/982916f8a60a4025a95b5a6c748e548c => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:42.167800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:42.170972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:42.186131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8323fe0-bcda-4875-8ff4-66e3534d66e7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:42.293944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a117c251-48f2-437e-8d95-4dbc99579f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.294457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a117c251-48f2-437e-8d95-4dbc99579f6e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.294748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a117c251-48f2-437e-8d95-4dbc99579f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.295052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a117c251-48f2-437e-8d95-4dbc99579f6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.295718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a117c251-48f2-437e-8d95-4dbc99579f6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:42.297340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a117c251-48f2-437e-8d95-4dbc99579f6e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:42.329275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3edea-1e22-4ece-bcb5-1c13bc4b236e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:42.334583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 966/1899] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:42.335980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8255ce-ffd4-4a80-a7f5-d9263984820f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.336274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8255ce-ffd4-4a80-a7f5-d9263984820f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.336540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8255ce-ffd4-4a80-a7f5-d9263984820f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.337195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.338604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.338994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.339353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.340095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:42.341934 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:42.355889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 934/1900] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:42.357042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:42.359986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:42.376729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f76c3-3cc1-4b89-9b99-13e494aff552 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:42.384147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.385136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.385670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.386180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] PATH_INFO: `/v3/projects/905cb70996ed4cf7afcc186293d7a51f/users/1fefdb1dd14949399ccad8fb442dbd28/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.397544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=83,keystone:INSERT=7,keystone:DELETE=10 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:42.407466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ea8255ce-ffd4-4a80-a7f5-d9263984820f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:42.414371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ea8255ce-ffd4-4a80-a7f5-d9263984820f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:42.423799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=905cb70996ed4cf7afcc186293d7a51f, user_id=1fefdb1dd14949399ccad8fb442dbd28, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:42.427049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:42.446985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 935/1901] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:42 2022] PUT /identity/v3/projects/905cb70996ed4cf7afcc186293d7a51f/users/1fefdb1dd14949399ccad8fb442dbd28/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:42.448218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:42.451271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 967/1902] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:42.460276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.460613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.460880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.525851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] MFA Rules not processed for user `1fefdb1dd14949399ccad8fb442dbd28`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:42.527408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:42.564762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:42.643336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 968/1903] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:42 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:42.644360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:42.649906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:42.651891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099e9ff6-1587-4e89-8a7e-9d59f47405b1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:42.669711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.671024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.675430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.676081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] PATH_INFO: `/v3/projects/534865bd729542edb652f63af856077e/users/b9519a147a004ba3a30b0c5eaa65aff4/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.682417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=534865bd729542edb652f63af856077e, user_id=b9519a147a004ba3a30b0c5eaa65aff4, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:42.686050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:42.714223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 936/1904] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:42 2022] PUT /identity/v3/projects/534865bd729542edb652f63af856077e/users/b9519a147a004ba3a30b0c5eaa65aff4/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:42.715107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:42.718227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:42.820373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe47f43-03fa-44d4-955e-e1490d5ee920 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:42.896075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.896735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.896985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.897620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:42.936065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cb6ae-6ccc-4cb6-9245-6b9e903f84e0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:42.950557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0887cb-c9f4-4195-a1f0-9600fddb431a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:42.951217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0887cb-c9f4-4195-a1f0-9600fddb431a service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:42.951503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0887cb-c9f4-4195-a1f0-9600fddb431a service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:42.951760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0887cb-c9f4-4195-a1f0-9600fddb431a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:43.079785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:43.083417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:43.093831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0887cb-c9f4-4195-a1f0-9600fddb431a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:43.102905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0887cb-c9f4-4195-a1f0-9600fddb431a service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:43.112209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 969/1905] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:43.115182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 937/1906] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:42 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:43.138489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:43.152192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25801a4-1c2d-45b0-b75d-5a0472ebbd25 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:43.162566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:43.163509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:43.163792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:43.164067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] PATH_INFO: `/v3/users/1fefdb1dd14949399ccad8fb442dbd28` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:43.165237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1fefdb1dd14949399ccad8fb442dbd28)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:43.167396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:43.225008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:43.247787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1fefdb1dd14949399ccad8fb442dbd28'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:43.257755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1fefdb1dd14949399ccad8fb442dbd28'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:43.281711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1fefdb1dd14949399ccad8fb442dbd28'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:43.282552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1fefdb1dd14949399ccad8fb442dbd28'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:43.292170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1fefdb1dd14949399ccad8fb442dbd28'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:43.300101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 970/1907] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:43 2022] DELETE /identity/v3/users/1fefdb1dd14949399ccad8fb442dbd28 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:43.306099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:43.306486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:43.306842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:43.338163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:43.339718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:43.385046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:43.442590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 938/1908] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:43.459660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:43.461232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:43.562786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd7e393-3b58-4903-9fc3-5c69b04a03e6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:43.570803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:43.572121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:43.572121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:43.572425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:43.585375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:43.586895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:43.599985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 971/1909] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:43.867521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:43.869900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:43.885231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169b172-6b51-49ef-a57d-8da89f23b1a5 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:43.902795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:43.904016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:43.905106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:43.905583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] PATH_INFO: `/v3/projects/5165845504364fbca1c38254cb657662` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:43.907290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5165845504364fbca1c38254cb657662)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:43.909943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:43.996571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5165845504364fbca1c38254cb657662', 'request_id': 'req-6f698bd6-190d-4fdf-b292-5c819674ee35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.006883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5165845504364fbca1c38254cb657662', 'request_id': 'req-6f698bd6-190d-4fdf-b292-5c819674ee35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.015771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5165845504364fbca1c38254cb657662', 'request_id': 'req-6f698bd6-190d-4fdf-b292-5c819674ee35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.040674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] The token cache is being invalidate because project 5165845504364fbca1c38254cb657662 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:44.041165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.042425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.045437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 939/1910] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:43 2022] DELETE /identity/v3/projects/5165845504364fbca1c38254cb657662 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:44.049160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.087449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d1a8d5-5d1b-4862-8ec9-921ef384e0b8 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.173279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.173937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.174212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.174480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] PATH_INFO: `/v3/users/0f0099efbb8f4d4895107ef89b717f13` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.177351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f0099efbb8f4d4895107ef89b717f13)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.179308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.249053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.250639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.270556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f0099efbb8f4d4895107ef89b717f13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.273155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:44.284748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f0099efbb8f4d4895107ef89b717f13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.289197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f698bd6-190d-4fdf-b292-5c819674ee35 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.295206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.296510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.296510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.304492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f0099efbb8f4d4895107ef89b717f13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.305396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f0099efbb8f4d4895107ef89b717f13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.310030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] PATH_INFO: `/v3/projects/905cb70996ed4cf7afcc186293d7a51f` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.310030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] RBAC: Authorizing `identity:delete_project(project_id=905cb70996ed4cf7afcc186293d7a51f)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.311563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.313453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4817fab-f760-4be5-8953-77d19e7c4c4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f0099efbb8f4d4895107ef89b717f13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.333096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 972/1911] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:44 2022] DELETE /identity/v3/users/0f0099efbb8f4d4895107ef89b717f13 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:44.334619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.334925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.340716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.399625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] MFA Rules not processed for user `b9519a147a004ba3a30b0c5eaa65aff4`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:44.400855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.421875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '905cb70996ed4cf7afcc186293d7a51f', 'request_id': 'req-18ec741e-228a-41ae-8820-725a6e25cefc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.439289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '905cb70996ed4cf7afcc186293d7a51f', 'request_id': 'req-18ec741e-228a-41ae-8820-725a6e25cefc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.449590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '905cb70996ed4cf7afcc186293d7a51f', 'request_id': 'req-18ec741e-228a-41ae-8820-725a6e25cefc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.476105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] The token cache is being invalidate because project 905cb70996ed4cf7afcc186293d7a51f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:44.476398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.478325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-18ec741e-228a-41ae-8820-725a6e25cefc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.480170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 940/1912] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:44 2022] DELETE /identity/v3/projects/905cb70996ed4cf7afcc186293d7a51f => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:44.489539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f7cf67-dfc7-4e41-b4a8-7ab6d3877f2e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.489963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f7cf67-dfc7-4e41-b4a8-7ab6d3877f2e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.490338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f7cf67-dfc7-4e41-b4a8-7ab6d3877f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.536825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-60f7cf67-dfc7-4e41-b4a8-7ab6d3877f2e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:44.538338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-60f7cf67-dfc7-4e41-b4a8-7ab6d3877f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.545919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 973/1913] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:44.553679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.557334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.617674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:44.652692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 941/1914] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:44.661644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.661936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.662228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.675739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae140a1-470b-463f-b0af-1dfab2fc994e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.683265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:44.684793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.685235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.685684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.686072 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.686447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.689162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.694732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 942/1915] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:44.698706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.708799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b857a0c0-ab4f-4cf3-94eb-445320df2036 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.713834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.714444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.714723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.715038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.715840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:44.716187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.717997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.735332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 943/1916] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:44.735332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.735332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.754830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da85bd0-f684-4a28-a170-899878a14c63 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.759951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.761298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.761413 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.761937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.762274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.762492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] PATH_INFO: `/v3/projects/9f996c33780940f882f82cdd8bdfc3d4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.763758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f996c33780940f882f82cdd8bdfc3d4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.765698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.770919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 974/1917] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:44 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:44.773378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.780398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=96 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:44.787083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de6bbae-e121-4ea5-b0d4-bb526b67c0d4 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.793301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.794006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.794346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.794725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.795520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.797612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.826064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f996c33780940f882f82cdd8bdfc3d4', 'request_id': 'req-b877e9b8-b00e-42c1-804f-0718574eeccf'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.829831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 975/1918] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/projects => generated 377 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:44.834420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.838113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f996c33780940f882f82cdd8bdfc3d4', 'request_id': 'req-b877e9b8-b00e-42c1-804f-0718574eeccf'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.859029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f996c33780940f882f82cdd8bdfc3d4', 'request_id': 'req-b877e9b8-b00e-42c1-804f-0718574eeccf'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.875751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] The token cache is being invalidate because project 9f996c33780940f882f82cdd8bdfc3d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:44.876292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.877928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b877e9b8-b00e-42c1-804f-0718574eeccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:44.880352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 944/1919] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:44 2022] DELETE /identity/v3/projects/9f996c33780940f882f82cdd8bdfc3d4 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:44.885702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d3c79d-e2ad-4afb-8c7d-b769afe20677 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.886219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d3c79d-e2ad-4afb-8c7d-b769afe20677 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.886682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d3c79d-e2ad-4afb-8c7d-b769afe20677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.909624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-99d3c79d-e2ad-4afb-8c7d-b769afe20677 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:44.911205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-99d3c79d-e2ad-4afb-8c7d-b769afe20677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.914735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1aaeb9-7660-46ea-bb75-6b06b848a0a1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.927307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 945/1920] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:44.927487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.927487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.927487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.927487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.927770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.929019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:44.929499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.929787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.930082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.955766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 976/1921] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/users => generated 289 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:44.958838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.960975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.967842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:44.969219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:44.976303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aae325-3071-4f0a-8ff4-9278c17d3f5f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:44.988705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 946/1922] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:44.992051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:44.992900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:44.993885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:44.995333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:44.996200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:44.996874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:44.998561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.008844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60880ea-f1bb-4dc4-9a68-f5b0f47a02d2 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.013391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 977/1923] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:44 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:45.017330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.018657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.019366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.019820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.020715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.021766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:45.022288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.024077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.037600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 947/1924] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:45.042456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f2598-da9f-49a6-904b-8311097cba1e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.042843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.055843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.056551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.056919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.057271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.062986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b10fef-2dcc-4c6a-ab3f-1952751c40c9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.064750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.067569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.069405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.069600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.069874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.070146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.070915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.072843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.086549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 978/1925] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:45 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:45.090923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.103359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624d60a-9701-4813-a41c-ea65b35b670b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.106850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 948/1926] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:45 2022] POST /identity/v3/projects => generated 383 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:45.108681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527af65a-1e1b-41e6-a53e-347e035a861a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.109529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527af65a-1e1b-41e6-a53e-347e035a861a admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.110514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527af65a-1e1b-41e6-a53e-347e035a861a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.110514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527af65a-1e1b-41e6-a53e-347e035a861a admin admin] PATH_INFO: `/v3/projects/a5c7d1ed6bc74e828c2bda14d1ab8485/users/861bd4018c6446a1985ad1a742262207/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.111470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.131093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527af65a-1e1b-41e6-a53e-347e035a861a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5c7d1ed6bc74e828c2bda14d1ab8485, user_id=861bd4018c6446a1985ad1a742262207, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.148211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527af65a-1e1b-41e6-a53e-347e035a861a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.175885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 979/1927] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:45 2022] PUT /identity/v3/projects/a5c7d1ed6bc74e828c2bda14d1ab8485/users/861bd4018c6446a1985ad1a742262207/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:45.185528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:45.187615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.187986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.188270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.203320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019c3f0-6eb2-45cc-891c-ce87ab09a886 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.239520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] MFA Rules not processed for user `861bd4018c6446a1985ad1a742262207`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:45.241041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:45.289785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.290462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.290754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.291107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.291755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.293880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.322748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 949/1928] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:45 2022] POST /identity/v3/users => generated 292 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:45.323772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.324618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 980/1929] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:45.335843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7086e9e-02ee-4485-a535-4eef4475df10 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.336003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.337908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:45.340843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.341598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.341882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.342172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.342819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.344475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.356969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 950/1930] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:45.357107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:45.360858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.371742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ce4db-7aa4-4b24-ab38-72c8dd3f2d5e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.376409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.377001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.377283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.377561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.378207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.380431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.390854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 951/1931] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:45.395039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.400131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beafb923-ab1f-4b6c-9c77-a0dec4961e4e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.403832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1accb850-ca25-4077-a9f1-59fa7479e6a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.404577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6829bc-a450-4934-a137-4c859d82a0de admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.404706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1accb850-ca25-4077-a9f1-59fa7479e6a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.404983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1accb850-ca25-4077-a9f1-59fa7479e6a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.405267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1accb850-ca25-4077-a9f1-59fa7479e6a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.409520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.410171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.410459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.410775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] PATH_INFO: `/v3/projects/b47715d37b5948d39cb9ea7e7dbf5eb0/users/2f242e3a9c3249378c0deaabc30e9108/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.414623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1accb850-ca25-4077-a9f1-59fa7479e6a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.415766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1accb850-ca25-4077-a9f1-59fa7479e6a0 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.426034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 981/1932] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:45.429487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b47715d37b5948d39cb9ea7e7dbf5eb0, user_id=2f242e3a9c3249378c0deaabc30e9108, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.432536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.446180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 952/1933] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:45 2022] PUT /identity/v3/projects/b47715d37b5948d39cb9ea7e7dbf5eb0/users/2f242e3a9c3249378c0deaabc30e9108/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:39:45.456587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.456947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.457229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.486796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] MFA Rules not processed for user `2f242e3a9c3249378c0deaabc30e9108`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:45.488071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:45.554402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 982/1934] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:45.570018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.572377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:45.687936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ac1557-a220-450d-b98d-531a026c44c8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.694363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c0d675-2213-4459-bf36-47c5a5966b49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.694363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c0d675-2213-4459-bf36-47c5a5966b49 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.694363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c0d675-2213-4459-bf36-47c5a5966b49 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.694363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c0d675-2213-4459-bf36-47c5a5966b49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.704362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c0d675-2213-4459-bf36-47c5a5966b49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.708170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c0d675-2213-4459-bf36-47c5a5966b49 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.727230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 953/1935] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:45 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:45.760291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:45.836739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362123e5-310e-4d6e-a6bf-97d8811e424a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:45.841175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:45.841817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.842110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.842380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] PATH_INFO: `/v3/users/2f242e3a9c3249378c0deaabc30e9108` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.843510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f242e3a9c3249378c0deaabc30e9108)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:45.845398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:45.890831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f242e3a9c3249378c0deaabc30e9108'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:45.893354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:45.899075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f242e3a9c3249378c0deaabc30e9108'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:45.913787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f242e3a9c3249378c0deaabc30e9108'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:45.914574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f242e3a9c3249378c0deaabc30e9108'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:45.921755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f242e3a9c3249378c0deaabc30e9108'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:45.928135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 983/1936] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:45 2022] DELETE /identity/v3/users/2f242e3a9c3249378c0deaabc30e9108 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:45.932950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:45.933321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:45.933597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:45.956201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:45.957534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:46.001192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:46.024540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 954/1937] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:46.035494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:46.038312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:46.105810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bd0600-7825-42da-ac33-6f666fdf74e2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:46.110769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24ab391-aa3e-43ae-aaff-e56274e64c0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:46.110769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24ab391-aa3e-43ae-aaff-e56274e64c0b service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:46.111104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24ab391-aa3e-43ae-aaff-e56274e64c0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:46.111381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24ab391-aa3e-43ae-aaff-e56274e64c0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:46.123252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24ab391-aa3e-43ae-aaff-e56274e64c0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:46.125049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24ab391-aa3e-43ae-aaff-e56274e64c0b service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:46.136203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 984/1938] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:46.716153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:46.717707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:46.730854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfae7e07-4484-4cee-bb47-35db6ddb31f9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:46.736348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:46.737096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:46.739387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:46.739776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] PATH_INFO: `/v3/projects/b47715d37b5948d39cb9ea7e7dbf5eb0` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:46.741082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b47715d37b5948d39cb9ea7e7dbf5eb0)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:46.743947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:46.864266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b47715d37b5948d39cb9ea7e7dbf5eb0', 'request_id': 'req-5d139d28-698f-4fd0-b90d-2139e8f6a321'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:46.867401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b47715d37b5948d39cb9ea7e7dbf5eb0', 'request_id': 'req-5d139d28-698f-4fd0-b90d-2139e8f6a321'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:46.873032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b47715d37b5948d39cb9ea7e7dbf5eb0', 'request_id': 'req-5d139d28-698f-4fd0-b90d-2139e8f6a321'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:46.890285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] The token cache is being invalidate because project b47715d37b5948d39cb9ea7e7dbf5eb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:46.891149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:46.892078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5d139d28-698f-4fd0-b90d-2139e8f6a321 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:46.900198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:46.900493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:46.900772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:46.914854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 955/1939] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:46 2022] DELETE /identity/v3/projects/b47715d37b5948d39cb9ea7e7dbf5eb0 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:46.929916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:46.931473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:47.005132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 985/1940] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:47.013671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.013959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.014297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.036733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:47.038094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:47.056835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 956/1941] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:39:47.060580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.079188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e4c4c8-4fb1-486a-8693-d2835f385210 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.086755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.087432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.087716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.087985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.088647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:47.088956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.090561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.106970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 986/1942] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:47.111177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.120431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871985d4-8b25-43ee-a39d-cab09a7db840 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.125293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.125880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.126149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.126421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.127150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.129028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.158790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 957/1943] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:47 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:47.163006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.174008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94414a97-5355-41f4-a93f-fbeb01d9a33f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.229383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:47.244534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.245408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.245954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.246468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.247622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.250602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.276613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 987/1944] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:47 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:47.280575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.290781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e2c48-4460-49b1-a3a8-264af79cf3e7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.295528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.296141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.296481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.296786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.297433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.299113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.315102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 958/1945] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:47.315191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.324918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ad8d43-3126-4e0e-866c-db332d9a5f9d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.329773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a99607-db2d-49a5-b9b5-e3da3e575f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.330383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a99607-db2d-49a5-b9b5-e3da3e575f12 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.330661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a99607-db2d-49a5-b9b5-e3da3e575f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.330965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a99607-db2d-49a5-b9b5-e3da3e575f12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.331615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a99607-db2d-49a5-b9b5-e3da3e575f12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.333846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a99607-db2d-49a5-b9b5-e3da3e575f12 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.350913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 988/1946] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:47 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:47.355582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.366254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38ba23-1eb6-474d-8f3e-1a6c526cdf28 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.371322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.372220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.372565 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.373019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] PATH_INFO: `/v3/projects/6471f685e5664252939289bec9191d3e/users/2ac50cf9666f433a98232eb034bbecb7/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.391581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6471f685e5664252939289bec9191d3e, user_id=2ac50cf9666f433a98232eb034bbecb7, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.394568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.408811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 959/1947] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:47 2022] PUT /identity/v3/projects/6471f685e5664252939289bec9191d3e/users/2ac50cf9666f433a98232eb034bbecb7/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:47.419191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.419767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.420056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.436606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.439939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:47.463031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] MFA Rules not processed for user `2ac50cf9666f433a98232eb034bbecb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:47.464281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:47.540653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafefbe-6ea0-4ff2-b762-8579107cabe6 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.546832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.547570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.547930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.548274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.550169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:47.566364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 989/1948] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:47.573873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:47.611297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:47.616772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:47.629327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.631835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.632963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b4264-db69-463d-86f0-e74dd5bed2e4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:47.638706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5280b50c-5638-433a-9f57-1080f8a56962 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:47.639862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280b50c-5638-433a-9f57-1080f8a56962 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:47.640071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280b50c-5638-433a-9f57-1080f8a56962 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:47.640524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280b50c-5638-433a-9f57-1080f8a56962 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:47.643409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 960/1949] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:47 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:39:47.650262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5280b50c-5638-433a-9f57-1080f8a56962 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:47.652355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5280b50c-5638-433a-9f57-1080f8a56962 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:47.663441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 990/1950] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:52.894338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:52.896503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:52.960300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee0d7b5-0a30-4ff9-b4b5-646eb1584442 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:52.964741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:52.965368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:52.965738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:52.966018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] PATH_INFO: `/v3/users/2319df8bb7c94393b41765cbe4072f4d` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:52.967187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2319df8bb7c94393b41765cbe4072f4d)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:52.969137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:53.013464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2319df8bb7c94393b41765cbe4072f4d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.020988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2319df8bb7c94393b41765cbe4072f4d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.032777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2319df8bb7c94393b41765cbe4072f4d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.033411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2319df8bb7c94393b41765cbe4072f4d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.040168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2319df8bb7c94393b41765cbe4072f4d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.046027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 961/1951] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:52 2022] DELETE /identity/v3/users/2319df8bb7c94393b41765cbe4072f4d => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:53.050392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:53.050907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:53.051028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:53.070524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:53.071752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:53.132270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 991/1952] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:53.142294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:53.143737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:53.185041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:53.210140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc1995-7a79-408f-9952-8d472163ea8c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:53.213866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:53.214529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:53.214810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:53.215178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:53.226671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:53.228141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:53.240412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 962/1953] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:53.539365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:53.540836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:53.553088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab92ef6f-cbef-45bb-a77b-83acc5954743 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:53.560545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:53.561323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:53.561736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:53.562055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] PATH_INFO: `/v3/projects/69966549f30d4d25ade9bdaa804da7ae` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:53.563328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=69966549f30d4d25ade9bdaa804da7ae)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:53.565558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:53.612886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:53.615472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69966549f30d4d25ade9bdaa804da7ae', 'request_id': 'req-a41482e5-44d6-4ff9-813b-d48f834ca1ce'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.624735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69966549f30d4d25ade9bdaa804da7ae', 'request_id': 'req-a41482e5-44d6-4ff9-813b-d48f834ca1ce'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.632217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69966549f30d4d25ade9bdaa804da7ae', 'request_id': 'req-a41482e5-44d6-4ff9-813b-d48f834ca1ce'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.644458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] The token cache is being invalidate because project 69966549f30d4d25ade9bdaa804da7ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:53.644778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.645461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.647187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 992/1954] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:53 2022] DELETE /identity/v3/projects/69966549f30d4d25ade9bdaa804da7ae => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:53.650852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:53.660685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a80b52c-4867-44fc-8920-41f2059b32bb service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:53.730199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:53.730855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:53.731168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:53.731498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] PATH_INFO: `/v3/users/2fa5026b16ed4d6a89c39613c345ec40` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:53.732616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fa5026b16ed4d6a89c39613c345ec40)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:53.734560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:53.783117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fa5026b16ed4d6a89c39613c345ec40'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.791617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fa5026b16ed4d6a89c39613c345ec40'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.806510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fa5026b16ed4d6a89c39613c345ec40'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.807229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fa5026b16ed4d6a89c39613c345ec40'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.814187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0d447eb0-4404-4625-b755-0ab288ce045e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fa5026b16ed4d6a89c39613c345ec40'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:53.819948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 963/1955] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:53 2022] DELETE /identity/v3/users/2fa5026b16ed4d6a89c39613c345ec40 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:54.088696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.090556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:54.106547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41482e5-44d6-4ff9-813b-d48f834ca1ce admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.164490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.165114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.165391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.165666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] PATH_INFO: `/v3/projects/6078f9184764475796b3c0e9bb413d45` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.166770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6078f9184764475796b3c0e9bb413d45)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.168694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.216077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6078f9184764475796b3c0e9bb413d45', 'request_id': 'req-07d2cfdf-31a5-40dc-9edf-529a0d845105'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:54.223126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6078f9184764475796b3c0e9bb413d45', 'request_id': 'req-07d2cfdf-31a5-40dc-9edf-529a0d845105'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:54.228779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6078f9184764475796b3c0e9bb413d45', 'request_id': 'req-07d2cfdf-31a5-40dc-9edf-529a0d845105'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:54.240811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] The token cache is being invalidate because project 6078f9184764475796b3c0e9bb413d45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:54.241101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:54.241759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-07d2cfdf-31a5-40dc-9edf-529a0d845105 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:54.243280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 993/1956] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:54 2022] DELETE /identity/v3/projects/6078f9184764475796b3c0e9bb413d45 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:54.248484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.248484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.248936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.275041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:54.276287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:54.328331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:54.338421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 964/1957] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:54.346301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.346585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.346858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.367335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:54.368575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:54.376360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 994/1958] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:54.381444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.391497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96108b4-81f0-474c-aea0-389b99b0a9ac None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.396475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.397098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.397377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.397650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.398309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:54.398620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.400260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.416491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 965/1959] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:39:54.417539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.427295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d69a8c-0606-4be5-832b-470ce6586f30 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.431969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.432566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.432848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.433120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.433755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.435689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.466983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 995/1960] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/projects => generated 387 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:54.468104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.478450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab87c90-9a0b-4c59-9c04-62f3495b8520 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.541660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.542192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.542475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.542759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.543448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.545370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.572428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 966/1961] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/users => generated 294 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:54.573760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.586064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61073150-bd79-4cea-ac8a-0ce665b16bb0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.593497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.594370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.595188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.595537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.596265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.597906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.610568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 996/1962] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:54.612274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.622676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e531e54-37fd-4f42-9e4c-a9d5276c0f32 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.627470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba89f46-1fd0-422d-861d-724e052162b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.628083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba89f46-1fd0-422d-861d-724e052162b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.628655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba89f46-1fd0-422d-861d-724e052162b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.628957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba89f46-1fd0-422d-861d-724e052162b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.630157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba89f46-1fd0-422d-861d-724e052162b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.632254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba89f46-1fd0-422d-861d-724e052162b2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.643597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 967/1963] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:54.648394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.658173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60370054-339e-432d-897f-bb6e403c4cc0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.662764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.663451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.663731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.664006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] PATH_INFO: `/v3/projects/799ba91964b64861a74b7795f79cbe2c/users/58f9f8f17007489899077d6e60b2b793/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.679876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=799ba91964b64861a74b7795f79cbe2c, user_id=58f9f8f17007489899077d6e60b2b793, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.682502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.702522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 997/1964] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:54 2022] PUT /identity/v3/projects/799ba91964b64861a74b7795f79cbe2c/users/58f9f8f17007489899077d6e60b2b793/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:54.713605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.713894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.714167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.742079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] MFA Rules not processed for user `58f9f8f17007489899077d6e60b2b793`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:54.743323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:54.810076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 968/1965] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:54.814822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.819873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:54.887108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6ec1a5-e8ad-4c71-b537-f1c279a9a4ac admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:54.892035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:54.892674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:54.892953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:54.893225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:54.893872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:54.895855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:54.927301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 998/1966] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/projects => generated 385 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:54.937750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:54.949529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e08ad0-787a-4c05-8a2b-29f66f94dd06 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:55.000579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:55.012527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:55.013150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.013480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.013893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.014552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:55.016648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:55.044846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 969/1967] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:54 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:55.046153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:55.057744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac11cb7-be33-4b5f-aba8-3143b1a51d52 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:55.063207 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:55.063770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.064304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.064595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.065260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:55.067648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:55.080245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 999/1968] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:55 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:55.085017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:55.098158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12de7ce8-e9df-4d2e-9dbd-c5f3c25a7ef4 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:55.102903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:55.103693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.103976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.104251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] PATH_INFO: `/v3/projects/cd8140c8b69240049c840041f26c92e2/users/f568a810550a446296bd280a431850c8/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.122865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd8140c8b69240049c840041f26c92e2, user_id=f568a810550a446296bd280a431850c8, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:55.125631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:55.141210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 970/1969] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:55 2022] PUT /identity/v3/projects/cd8140c8b69240049c840041f26c92e2/users/f568a810550a446296bd280a431850c8/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:55.145531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:55.155997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a48786-61aa-4326-84e0-bdad22e90176 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:55.214001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadf9ed2-8c23-4764-a2eb-b5238d4e259a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:55.214609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadf9ed2-8c23-4764-a2eb-b5238d4e259a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.214887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadf9ed2-8c23-4764-a2eb-b5238d4e259a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.215186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadf9ed2-8c23-4764-a2eb-b5238d4e259a admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.215826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadf9ed2-8c23-4764-a2eb-b5238d4e259a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:55.217391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadf9ed2-8c23-4764-a2eb-b5238d4e259a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:55.234250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1000/1970] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:55 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:55.235516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:55.247504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b264fa-5210-4e88-93ce-71e2b7092eaf admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:55.252098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:55.252754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.253037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.253314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] PATH_INFO: `/v3/projects/cd8140c8b69240049c840041f26c92e2/users/f568a810550a446296bd280a431850c8/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.255387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd8140c8b69240049c840041f26c92e2, user_id=f568a810550a446296bd280a431850c8, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:55.262239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:55.271468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 971/1971] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:55 2022] PUT /identity/v3/projects/cd8140c8b69240049c840041f26c92e2/users/f568a810550a446296bd280a431850c8/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:55.281836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.282125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.282796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.311406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] MFA Rules not processed for user `f568a810550a446296bd280a431850c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:55.312608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:55.374364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:55.383088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1001/1972] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:55 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:55.390522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:55.392253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:55.451703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfce16c2-6771-49ca-a3b9-d47ff87e3865 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:55.455522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a45c068-4bcf-4e62-af92-30bd25cc7d70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:55.456136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a45c068-4bcf-4e62-af92-30bd25cc7d70 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:55.456416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a45c068-4bcf-4e62-af92-30bd25cc7d70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:55.456690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a45c068-4bcf-4e62-af92-30bd25cc7d70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:55.465752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a45c068-4bcf-4e62-af92-30bd25cc7d70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:55.467211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a45c068-4bcf-4e62-af92-30bd25cc7d70 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:55.477311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 972/1973] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:55 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:56.159671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:56.227572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75376940-7025-4ba8-a616-73bebfe3629b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:56.233746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:56.234397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:56.234678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:56.234974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] PATH_INFO: `/v3/users/58f9f8f17007489899077d6e60b2b793` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:56.236124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58f9f8f17007489899077d6e60b2b793)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:56.238190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:56.288667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58f9f8f17007489899077d6e60b2b793'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.295747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58f9f8f17007489899077d6e60b2b793'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.312360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58f9f8f17007489899077d6e60b2b793'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.312993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58f9f8f17007489899077d6e60b2b793'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.320856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58f9f8f17007489899077d6e60b2b793'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.326915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1002/1974] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:56 2022] DELETE /identity/v3/users/58f9f8f17007489899077d6e60b2b793 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:56.331272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:56.331783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:56.331885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:56.352371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:56.353583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:56.414786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 973/1975] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:56.429673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:56.429673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:56.489733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b706ad4-be77-4165-b660-5c3d1153e3b2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:56.493417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:56.494033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:56.494320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:56.494603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:56.504384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:56.505804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:56.515415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1003/1976] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:56.788137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:56.789872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:56.803266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac8c7fb-70ce-4c28-9f48-780a8ac40003 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:56.808003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:56.808636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:56.808923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:56.809352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] PATH_INFO: `/v3/projects/799ba91964b64861a74b7795f79cbe2c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:56.810500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=799ba91964b64861a74b7795f79cbe2c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:56.812501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:56.855573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:56.859536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '799ba91964b64861a74b7795f79cbe2c', 'request_id': 'req-6da290d4-f942-44e6-91a7-57fcbe5651ec'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.867761 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '799ba91964b64861a74b7795f79cbe2c', 'request_id': 'req-6da290d4-f942-44e6-91a7-57fcbe5651ec'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.873938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '799ba91964b64861a74b7795f79cbe2c', 'request_id': 'req-6da290d4-f942-44e6-91a7-57fcbe5651ec'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.884887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] The token cache is being invalidate because project 799ba91964b64861a74b7795f79cbe2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:56.885167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.885809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:56.887380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 974/1977] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:56 2022] DELETE /identity/v3/projects/799ba91964b64861a74b7795f79cbe2c => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:56.891250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:56.900860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fbccb-5664-4808-8c7b-0f2f4c78cf61 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:56.918967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:56.963550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:56.964402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:56.964692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:56.965001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] PATH_INFO: `/v3/users/f568a810550a446296bd280a431850c8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:56.966125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f568a810550a446296bd280a431850c8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:56.968153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.031417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f568a810550a446296bd280a431850c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.040952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f568a810550a446296bd280a431850c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.055417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f568a810550a446296bd280a431850c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.056249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f568a810550a446296bd280a431850c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.063457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e1106dfc-3639-4fff-9f49-095daba38c81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f568a810550a446296bd280a431850c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.071188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1004/1978] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:56 2022] DELETE /identity/v3/users/f568a810550a446296bd280a431850c8 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:57.360442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.361997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:57.375040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da290d4-f942-44e6-91a7-57fcbe5651ec admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.432320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.432874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.433157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.433433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] PATH_INFO: `/v3/projects/cd8140c8b69240049c840041f26c92e2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.434553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd8140c8b69240049c840041f26c92e2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.436541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.485463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd8140c8b69240049c840041f26c92e2', 'request_id': 'req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.492811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd8140c8b69240049c840041f26c92e2', 'request_id': 'req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.498408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd8140c8b69240049c840041f26c92e2', 'request_id': 'req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.510015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] The token cache is being invalidate because project cd8140c8b69240049c840041f26c92e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:39:57.510278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.511108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c2e9ddf1-005a-492a-9839-b297cf5fdb2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:57.515011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 975/1979] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:57 2022] DELETE /identity/v3/projects/cd8140c8b69240049c840041f26c92e2 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:57.517277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.517762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.517934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.538367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:57.539611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:57.597403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1005/1980] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:57.604956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.605245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.605530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.624706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:57.625910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:57.634986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 976/1981] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:57.639205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.649438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695fde3b-35cd-4dc6-b5f2-a9a6f706d27b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.653997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.654595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.654874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.655181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.655825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:39:57.656130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.657720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.669028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1006/1982] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:39:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:39:57.673234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.682575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bfaf38-fbc3-483a-8095-5ac714a268d6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.687262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.687856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.688135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.688407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.689301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.691227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.718970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 977/1983] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:57 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:39:57.722925 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.732763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdccf264-29bc-497a-a6d8-8ce0338eeb74 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.756932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:57.791873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.792489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.792769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.793041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.793677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.795625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.817646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1007/1984] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:57 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:57.821440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.831724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65bb150-11f9-4f0b-a948-5f36448b06d9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.837226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.837835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.838230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.838512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.839206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.840827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.851105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 978/1985] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:57.855151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.865767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee389a68-2f1d-444b-9f2e-132a704b2b78 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.870341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54918e9b-66f1-4458-93c3-10d742061513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.871188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54918e9b-66f1-4458-93c3-10d742061513 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.871891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54918e9b-66f1-4458-93c3-10d742061513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.871891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54918e9b-66f1-4458-93c3-10d742061513 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.872501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54918e9b-66f1-4458-93c3-10d742061513 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.874117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54918e9b-66f1-4458-93c3-10d742061513 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.890154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1008/1986] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:39:57.891387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:57.901406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864c14ac-d8ba-49fa-b73b-bbbd5f31bca7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:57.906220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:57.906872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.907179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.907453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] PATH_INFO: `/v3/projects/ca33d7a6988d4c7c8c299be5e675c51e/users/a56ebc2572154b17a33fd078bfc5e27b/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.922701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca33d7a6988d4c7c8c299be5e675c51e, user_id=a56ebc2572154b17a33fd078bfc5e27b, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:57.925387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:57.934631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:57.942539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 979/1987] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:57 2022] PUT /identity/v3/projects/ca33d7a6988d4c7c8c299be5e675c51e/users/a56ebc2572154b17a33fd078bfc5e27b/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:57.954469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:57.954774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:57.955096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:57.984756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] MFA Rules not processed for user `a56ebc2572154b17a33fd078bfc5e27b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:57.986092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:58.089832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1009/1988] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:57 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:58.095690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.172305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51121f-4fa4-4112-946d-383126b11149 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.177281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.177926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.178210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.178470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.179163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.181209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.215789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 980/1989] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:39:58 2022] POST /identity/v3/projects => generated 381 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:58.220712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.233250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2137340-a4a3-4e80-88a1-6fc8b33f41d9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.313298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.314551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.315008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.315466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.316260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.318890 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.343535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1010/1990] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:39:58 2022] POST /identity/v3/users => generated 297 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:39:58.347937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.359203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b9fe0e-3ddd-4040-8571-cf65c1f4c230 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.364459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.365080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.365477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.365819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.366701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.368729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.383567 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 981/1991] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:58.385026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.396840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31275a5f-6bfe-49af-9c89-70fc29ce1f80 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.401674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.402338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.402622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.403050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] PATH_INFO: `/v3/projects/97ed0f91c9f14da3a5c14a2b195cf8b1/users/402cabf86b3341978be011cc153fa144/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.419771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97ed0f91c9f14da3a5c14a2b195cf8b1, user_id=402cabf86b3341978be011cc153fa144, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.424286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.445314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:58.451485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1011/1992] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:58 2022] PUT /identity/v3/projects/97ed0f91c9f14da3a5c14a2b195cf8b1/users/402cabf86b3341978be011cc153fa144/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:58.455378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.467611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2775daa8-7ba4-45fe-a37e-55f8f30db1e9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.537938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b49f020-21f8-49c9-ae9d-addb2ec3066f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.538667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b49f020-21f8-49c9-ae9d-addb2ec3066f admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.539064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b49f020-21f8-49c9-ae9d-addb2ec3066f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.539411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b49f020-21f8-49c9-ae9d-addb2ec3066f admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.540133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b49f020-21f8-49c9-ae9d-addb2ec3066f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.541810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b49f020-21f8-49c9-ae9d-addb2ec3066f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.552480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 982/1993] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:39:58 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:39:58.556632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.569828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe2d44-c9d0-4c19-b4f3-6abeaa4216f9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.576487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.577529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.577529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.577729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] PATH_INFO: `/v3/projects/97ed0f91c9f14da3a5c14a2b195cf8b1/users/402cabf86b3341978be011cc153fa144/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.579826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97ed0f91c9f14da3a5c14a2b195cf8b1, user_id=402cabf86b3341978be011cc153fa144, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.583240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.604498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1012/1994] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:39:58 2022] PUT /identity/v3/projects/97ed0f91c9f14da3a5c14a2b195cf8b1/users/402cabf86b3341978be011cc153fa144/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:58.607166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.607515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.607802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.642839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] MFA Rules not processed for user `402cabf86b3341978be011cc153fa144`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:58.644132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:58.708958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:58.720345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 983/1995] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:58 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:58.731684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:58.733392 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:58.800416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621ee0b9-f035-4efb-aedf-6cc6505ba718 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:58.804595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4dbac0-4294-4b37-91f0-7156f31f50ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:58.805206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4dbac0-4294-4b37-91f0-7156f31f50ed service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:58.805484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4dbac0-4294-4b37-91f0-7156f31f50ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:58.805755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4dbac0-4294-4b37-91f0-7156f31f50ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:58.814970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4dbac0-4294-4b37-91f0-7156f31f50ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:58.817122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4dbac0-4294-4b37-91f0-7156f31f50ed service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:58.826435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1013/1996] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:58 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:59.229303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:59.323126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158d7852-dafd-4803-bdfe-f0aaf77fac56 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:59.323126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:59.323126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:59.323126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:59.323730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] PATH_INFO: `/v3/users/a56ebc2572154b17a33fd078bfc5e27b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:59.323730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a56ebc2572154b17a33fd078bfc5e27b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:59.324469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:59.383946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a56ebc2572154b17a33fd078bfc5e27b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.392277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a56ebc2572154b17a33fd078bfc5e27b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.407309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a56ebc2572154b17a33fd078bfc5e27b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.408081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a56ebc2572154b17a33fd078bfc5e27b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.417239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a56ebc2572154b17a33fd078bfc5e27b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.423955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 984/1997] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:59 2022] DELETE /identity/v3/users/a56ebc2572154b17a33fd078bfc5e27b => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:59.428158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:59.428560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:59.428934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:59.448980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:59.450206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:59.531018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1014/1998] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:59.539046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:59.540683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:59.599201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31122c2-25d8-4423-ab14-1e9cee8720a1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:59.602745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef0aad0-6db2-4c28-8b28-f6a7aeb92d00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:59.603385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef0aad0-6db2-4c28-8b28-f6a7aeb92d00 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:59.603660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef0aad0-6db2-4c28-8b28-f6a7aeb92d00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:59.603956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef0aad0-6db2-4c28-8b28-f6a7aeb92d00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:59.613085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef0aad0-6db2-4c28-8b28-f6a7aeb92d00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:59.614510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef0aad0-6db2-4c28-8b28-f6a7aeb92d00 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:59.624364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 985/1999] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:39:59.723057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:59.724641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:59.737187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a01b21-8b99-43e3-82e8-aa6b23b014ea None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:59.742802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:59.743561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:59.743842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:59.744120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] PATH_INFO: `/v3/users/2ac50cf9666f433a98232eb034bbecb7` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:59.745430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ac50cf9666f433a98232eb034bbecb7)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:39:59.747371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:39:59.808464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ac50cf9666f433a98232eb034bbecb7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.819007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ac50cf9666f433a98232eb034bbecb7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.837231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ac50cf9666f433a98232eb034bbecb7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.838174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ac50cf9666f433a98232eb034bbecb7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.847086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ac50cf9666f433a98232eb034bbecb7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:39:59.854805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1015/2000] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:39:59 2022] DELETE /identity/v3/users/2ac50cf9666f433a98232eb034bbecb7 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:39:59.858999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:59.859550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:59.859550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:39:59.880366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:39:59.881580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:59.897053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:59.949971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:59.951501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:59.965150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 986/2001] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:39:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:39:59.971485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:39:59.973310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:39:59.977401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:39:59.990847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb593e-bfa6-49cb-ab7b-05aada35bee2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:39:59.997937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:39:59.999039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:39:59.999411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:39:59.999755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] PATH_INFO: `/v3/projects/ca33d7a6988d4c7c8c299be5e675c51e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:00.001342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca33d7a6988d4c7c8c299be5e675c51e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:00.003670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:00.041641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c06fd7-909b-42c0-830f-c79b5cf856f8 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:00.055998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca33d7a6988d4c7c8c299be5e675c51e', 'request_id': 'req-408cb6ff-4bca-4258-a905-499639d4a15a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.065067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca33d7a6988d4c7c8c299be5e675c51e', 'request_id': 'req-408cb6ff-4bca-4258-a905-499639d4a15a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.072350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca33d7a6988d4c7c8c299be5e675c51e', 'request_id': 'req-408cb6ff-4bca-4258-a905-499639d4a15a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.089672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] The token cache is being invalidate because project ca33d7a6988d4c7c8c299be5e675c51e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:00.089992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.091006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.092778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1016/2002] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:39:59 2022] DELETE /identity/v3/projects/ca33d7a6988d4c7c8c299be5e675c51e => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:00.096975 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:00.098459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:00.098746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:00.099391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:00.099669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:00.099948 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:00.101667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:00.213694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408cb6ff-4bca-4258-a905-499639d4a15a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:00.223179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:00.223851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:00.224707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:00.225024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] PATH_INFO: `/v3/users/402cabf86b3341978be011cc153fa144` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:00.226298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] RBAC: Authorizing `identity:delete_user(user_id=402cabf86b3341978be011cc153fa144)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:00.228932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:00.248916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:00.250778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:00.271741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 987/2003] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:39:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:40:00.331280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '402cabf86b3341978be011cc153fa144'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.342511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '402cabf86b3341978be011cc153fa144'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.375069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '402cabf86b3341978be011cc153fa144'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.376016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '402cabf86b3341978be011cc153fa144'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.385863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '402cabf86b3341978be011cc153fa144'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:00.397277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1017/2004] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:00 2022] DELETE /identity/v3/users/402cabf86b3341978be011cc153fa144 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:00.790407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:00.792889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:00.819216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce124ac-a64e-428d-8d2b-c55073594483 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:00.855144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:01.022629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.024461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.025186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.033238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] PATH_INFO: `/v3/projects/6471f685e5664252939289bec9191d3e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.040158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6471f685e5664252939289bec9191d3e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.045545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.067453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.071131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:01.117706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:01.143569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af36d528-a80f-4239-93db-e7dc59f3bceb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.162117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6471f685e5664252939289bec9191d3e', 'request_id': 'req-8ea73cbf-719c-4c20-9005-db74156147f1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.177753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6471f685e5664252939289bec9191d3e', 'request_id': 'req-8ea73cbf-719c-4c20-9005-db74156147f1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.187956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6471f685e5664252939289bec9191d3e', 'request_id': 'req-8ea73cbf-719c-4c20-9005-db74156147f1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.217969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] The token cache is being invalidate because project 6471f685e5664252939289bec9191d3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:01.217969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.219194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-8ea73cbf-719c-4c20-9005-db74156147f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.224320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 988/2005] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:00 2022] DELETE /identity/v3/projects/6471f685e5664252939289bec9191d3e => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:01.229103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f35d12-a94c-4ae9-b21c-a1647f3a1567 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.229388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f35d12-a94c-4ae9-b21c-a1647f3a1567 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.229663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f35d12-a94c-4ae9-b21c-a1647f3a1567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.280939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d8f35d12-a94c-4ae9-b21c-a1647f3a1567 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:01.280939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d8f35d12-a94c-4ae9-b21c-a1647f3a1567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:01.314676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.315310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.315585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.316552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] PATH_INFO: `/v3/projects/97ed0f91c9f14da3a5c14a2b195cf8b1` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.318430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97ed0f91c9f14da3a5c14a2b195cf8b1)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.320380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.465190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97ed0f91c9f14da3a5c14a2b195cf8b1', 'request_id': 'req-b4629bcd-e5cf-4c36-b218-37db4b4f3457'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.467467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 989/2006] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:01.473498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.473984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.474488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.478817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97ed0f91c9f14da3a5c14a2b195cf8b1', 'request_id': 'req-b4629bcd-e5cf-4c36-b218-37db4b4f3457'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.489567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97ed0f91c9f14da3a5c14a2b195cf8b1', 'request_id': 'req-b4629bcd-e5cf-4c36-b218-37db4b4f3457'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.510628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] The token cache is being invalidate because project 97ed0f91c9f14da3a5c14a2b195cf8b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:01.510945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.511735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b4629bcd-e5cf-4c36-b218-37db4b4f3457 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:01.514202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1018/2007] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:01 2022] DELETE /identity/v3/projects/97ed0f91c9f14da3a5c14a2b195cf8b1 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:01.526739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:01.529092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:01.536061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9063f34-75bf-44a5-92c0-f5b5c100cd30 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.536364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9063f34-75bf-44a5-92c0-f5b5c100cd30 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.536654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9063f34-75bf-44a5-92c0-f5b5c100cd30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.599684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f9063f34-75bf-44a5-92c0-f5b5c100cd30 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:01.601376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f9063f34-75bf-44a5-92c0-f5b5c100cd30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:01.651071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:01.678911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 990/2008] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:01.688634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.719393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b70aa-c04b-46e9-b354-a601ee55e1d5 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.730286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.731235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.731584 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.731924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.733313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:01.733620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.735554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.757610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 991/2009] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:40:01.758758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.761153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1019/2010] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:01.768702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.768990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.769971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c15272-b9aa-4890-b54b-d650017330e2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.770150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.774865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.775427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.775708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.776027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.776659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.791612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.792182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:01.793080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:01.803519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1020/2011] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:01.807749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.810974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 992/2012] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/projects => generated 359 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:01.812918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.817853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7764f3e-3c91-4024-9cbf-d83722b206b5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.845224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:01.884127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7ff7e-7e6a-495b-a917-7d6b7175e708 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.885612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.886302 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.886587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.886924 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.887534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:01.887838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.889279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.889499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.890047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.890517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.890802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.895276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.897604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.905639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1021/2013] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:40:01.910729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.925262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31dc330-832a-4fc9-81ea-37cd069f162a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.933302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 993/2014] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/users => generated 280 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:01.935611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.936359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.936707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.937083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.937338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.938590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.940979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.949825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ca028-bc96-4d8a-b17a-d38ed2c85beb admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:01.956540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:01.957403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:01.957949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:01.958243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:01.958997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:01.961194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:01.976403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1022/2015] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/projects => generated 385 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:01.976704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 994/2016] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:01 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:01.983646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.984335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:01.995653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f64fc7-d2eb-4f5d-82c3-e82749afb7fa admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.069262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.070255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.070542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.070879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.071801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.073687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.090910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 995/2017] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:01 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:02.091176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2193f5bd-5ab5-4b73-a573-cadbc04ee678 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.092769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.095529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.096480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.096879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.097375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.098806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.100860 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.110070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de109315-6ef7-438d-a6d5-885189c55d81 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.115576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dc4afd-47ab-4fb3-a079-601087ccc904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.116243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dc4afd-47ab-4fb3-a079-601087ccc904 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.116526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dc4afd-47ab-4fb3-a079-601087ccc904 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.116892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dc4afd-47ab-4fb3-a079-601087ccc904 admin admin] PATH_INFO: `/v3/projects/31e724e7a35e4e18aae599c27ab0f2ba/users/38785c03b052415db8f8c0355454ed84/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.139701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1023/2018] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:01 2022] POST /identity/v3/users => generated 293 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:02.142957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dc4afd-47ab-4fb3-a079-601087ccc904 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31e724e7a35e4e18aae599c27ab0f2ba, user_id=38785c03b052415db8f8c0355454ed84, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.146212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.146830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dc4afd-47ab-4fb3-a079-601087ccc904 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.159095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74565def-a2c9-423b-ad6c-00f0064d5791 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.163361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 996/2019] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:02 2022] PUT /identity/v3/projects/31e724e7a35e4e18aae599c27ab0f2ba/users/38785c03b052415db8f8c0355454ed84/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:02.177474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32063cba-4b97-44e6-8f95-7c7c66092404 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.178302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32063cba-4b97-44e6-8f95-7c7c66092404 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.178896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32063cba-4b97-44e6-8f95-7c7c66092404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.211342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:02.262168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-32063cba-4b97-44e6-8f95-7c7c66092404 None None] MFA Rules not processed for user `38785c03b052415db8f8c0355454ed84`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:02.264173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-32063cba-4b97-44e6-8f95-7c7c66092404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:02.270450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.271250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.271610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.275603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.276637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.278188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.317292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1024/2020] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:02 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:02.320062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.344051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366bbb01-032b-47a1-9e88-f25c6c89b916 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.367023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.367688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.367977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.368253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.368908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.371004 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.387797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1025/2021] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:02 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:02.390762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.413495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb24b847-af65-4414-8e14-72c5056443ff admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.422487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.423209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.423492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.423772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] PATH_INFO: `/v3/projects/0d979aea8ece48538a3e6ff9bebe394c/users/97c7dd71871f4689b01fd7433b8c8d20/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.434853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 997/2022] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:02 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:02.444746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:02.448701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d979aea8ece48538a3e6ff9bebe394c, user_id=97c7dd71871f4689b01fd7433b8c8d20, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.456308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.493291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1026/2023] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:02 2022] PUT /identity/v3/projects/0d979aea8ece48538a3e6ff9bebe394c/users/97c7dd71871f4689b01fd7433b8c8d20/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:02.509946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.510475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.511034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.582796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.583967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:02.591649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] MFA Rules not processed for user `97c7dd71871f4689b01fd7433b8c8d20`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:02.594257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:02.691013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb03ea-fda4-4f02-9810-3c350bdaf5f6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.697299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.698358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.698740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.699153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.701193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 998/2024] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:02.706573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.815589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.817164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.844309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1027/2025] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:02 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:02.856877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d87abc9-6515-409f-9235-1fc4a91f9199 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.864076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.864724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.865003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.865326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.866028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.867119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.874180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.912036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6631de-ddd3-4301-aff2-fda125ed1c4d service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:02.921293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:02.923203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:02.932276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:02.932276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] PATH_INFO: `/v3/users/38785c03b052415db8f8c0355454ed84` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:02.934786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 999/2026] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:02 2022] POST /identity/v3/projects => generated 385 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:02.943223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:02.943518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=38785c03b052415db8f8c0355454ed84)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:02.948613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:02.952382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:03.028764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38785c03b052415db8f8c0355454ed84'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.040967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38785c03b052415db8f8c0355454ed84'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.042716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd14dd0a-1a25-40ba-9656-54ac87fa6590 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.048868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.049507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.049917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.050205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.051446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.053371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.057893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38785c03b052415db8f8c0355454ed84'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.062564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38785c03b052415db8f8c0355454ed84'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.070695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c0df52c-f6d1-4296-813c-67d065dfcc1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38785c03b052415db8f8c0355454ed84'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.081454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1028/2027] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:02 2022] DELETE /identity/v3/users/38785c03b052415db8f8c0355454ed84 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:03.083519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1000/2028] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:02 2022] POST /identity/v3/users => generated 293 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:03.083640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.083804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.084087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.092475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.094375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.104946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:03.106101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.108976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca3e1d3-6408-41e0-a7ee-42b126ba2a13 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.118746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:03.172031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.173122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.173445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.173723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.174245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1029/2029] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:03.174370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.176235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.185363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.186861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.187036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1001/2030] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:03 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:03.191885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.200689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5addf66-d451-4423-8730-7e208deeda58 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.205494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.206125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.206402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.206674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.208024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.209630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.220423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1002/2031] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:03.225000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.235041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773d0430-2580-455e-b34d-3d6a422910e4 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.240132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4097a5-5831-45f4-b25d-83eab5f182ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.240792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4097a5-5831-45f4-b25d-83eab5f182ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.241077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4097a5-5831-45f4-b25d-83eab5f182ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.241359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4097a5-5831-45f4-b25d-83eab5f182ba admin admin] PATH_INFO: `/v3/projects/8d9082267e134b17b17641c19acbab5e/users/7f4d578ae4384d2fbd3bd2f8689be086/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.255183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b6fe5-8a09-4b74-958c-15c84b7cb8b6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.256388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4097a5-5831-45f4-b25d-83eab5f182ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d9082267e134b17b17641c19acbab5e, user_id=7f4d578ae4384d2fbd3bd2f8689be086, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.259196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4097a5-5831-45f4-b25d-83eab5f182ba admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.266438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.267172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.267453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.267722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.274622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1003/2032] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:03 2022] PUT /identity/v3/projects/8d9082267e134b17b17641c19acbab5e/users/7f4d578ae4384d2fbd3bd2f8689be086/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:03.281567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.283002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.284724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.285142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.285968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.294008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1030/2033] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:40:03.314663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] MFA Rules not processed for user `7f4d578ae4384d2fbd3bd2f8689be086`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:03.315922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.373110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:03.384185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1004/2034] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:03.388735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.399476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e83180b-30cf-42ba-b8f6-44f0799cd20e service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.463406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.464005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.464289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.464559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.465197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.467212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.496943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1031/2035] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:03 2022] POST /identity/v3/projects => generated 383 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:03.498300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.509942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec0fdaf-4cc0-454c-a5f7-98d66e1fd049 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.572108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.575147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.576337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.577075 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.577406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.588842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.590330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.595166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.595654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b349180f-0ade-483d-beea-8bccc5eb9d8f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.602344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.603474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.603825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.604226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] PATH_INFO: `/v3/projects/31e724e7a35e4e18aae599c27ab0f2ba` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.607159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31e724e7a35e4e18aae599c27ab0f2ba)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.609594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.622909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1005/2036] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:03 2022] POST /identity/v3/users => generated 298 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:03.624581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.639379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3acdf6b-57ea-456b-8bba-a25a8eb8b959 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.646512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.646985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.647359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.647662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.648336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.649932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.660668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:03.661656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1006/2037] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:03 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:03.665894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.671617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31e724e7a35e4e18aae599c27ab0f2ba', 'request_id': 'req-44a07b68-f5dc-497e-8a37-ef6d432f5d22'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.676811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162611e1-fc39-418f-bc77-16c21b134577 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.680441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31e724e7a35e4e18aae599c27ab0f2ba', 'request_id': 'req-44a07b68-f5dc-497e-8a37-ef6d432f5d22'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.687907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31e724e7a35e4e18aae599c27ab0f2ba', 'request_id': 'req-44a07b68-f5dc-497e-8a37-ef6d432f5d22'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.702119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] The token cache is being invalidate because project 31e724e7a35e4e18aae599c27ab0f2ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:03.702415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.703113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-44a07b68-f5dc-497e-8a37-ef6d432f5d22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:03.704878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1032/2038] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:03 2022] DELETE /identity/v3/projects/31e724e7a35e4e18aae599c27ab0f2ba => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:03.709979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd1c2ab-09f5-4367-a72d-0eb69ed2955f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.710560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd1c2ab-09f5-4367-a72d-0eb69ed2955f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.710560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd1c2ab-09f5-4367-a72d-0eb69ed2955f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.732826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5cd1c2ab-09f5-4367-a72d-0eb69ed2955f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:03.734752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5cd1c2ab-09f5-4367-a72d-0eb69ed2955f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.746395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.747104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.747469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.747773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] PATH_INFO: `/v3/projects/a6564a4b91484feb8e8cc067761a2a3d/users/7aa8ed109a3b44ea9cc263a2c14d0d77/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.768389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6564a4b91484feb8e8cc067761a2a3d, user_id=7aa8ed109a3b44ea9cc263a2c14d0d77, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.772774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.790900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1007/2039] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:03 2022] PUT /identity/v3/projects/a6564a4b91484feb8e8cc067761a2a3d/users/7aa8ed109a3b44ea9cc263a2c14d0d77/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:03.794889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.796936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.809388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1033/2040] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:03.816907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfa5f99-0d4d-49d3-a876-73ba7df0db7d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.817271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfa5f99-0d4d-49d3-a876-73ba7df0db7d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.817622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfa5f99-0d4d-49d3-a876-73ba7df0db7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.844107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-acfa5f99-0d4d-49d3-a876-73ba7df0db7d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:03.847278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-acfa5f99-0d4d-49d3-a876-73ba7df0db7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:03.892181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:03.901799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1758a99-73b4-418d-bffe-44f671d00de5 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.912298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.913086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.913491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.913902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.914856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.917799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.932397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1008/2041] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:03 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:03.937940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:03.967020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c6a07-29e9-4d66-8c56-153f3f9f4cdf admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:03.975247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:03.975924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:03.976203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:03.976472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] PATH_INFO: `/v3/projects/a6564a4b91484feb8e8cc067761a2a3d/users/7aa8ed109a3b44ea9cc263a2c14d0d77/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:03.978426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6564a4b91484feb8e8cc067761a2a3d, user_id=7aa8ed109a3b44ea9cc263a2c14d0d77, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:03.981902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:03.997911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1009/2042] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:03 2022] PUT /identity/v3/projects/a6564a4b91484feb8e8cc067761a2a3d/users/7aa8ed109a3b44ea9cc263a2c14d0d77/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:03.999032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1034/2043] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:04.003131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.012193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.013125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.013731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.049758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] MFA Rules not processed for user `7aa8ed109a3b44ea9cc263a2c14d0d77`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:04.051014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:04.083515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742d904b-0fe6-46a2-b0ed-4fd784d5387a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.090280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.091120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.091530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.091929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.092707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:04.093157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.094897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.117631 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1010/2044] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:40:04.120311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:04.122235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.136205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba57d032-7e39-4413-b8e6-1fa938700eb5 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.138445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1035/2045] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:04 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:04.143982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.146352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:04.148110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.149031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.149456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.149877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.150678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.156345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.214953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97164e3-010a-4fe9-8ee7-889d8b05e437 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.218943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1011/2046] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:04 2022] POST /identity/v3/projects => generated 369 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:04.220784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.274303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.274908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.275229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.275508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.277252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:04.308971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc42dacf-a543-416d-a6d7-181036939373 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.314627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.315267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.319091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.319386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.320042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.322020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.339637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.341058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.347182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1012/2047] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:04 2022] POST /identity/v3/users => generated 285 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:04.348718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.351028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1036/2048] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:40:04.358399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2facc8df-9b40-42d7-afe9-25174a6b2187 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.363056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.364241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.364525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.364801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.365473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.367105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.377068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1013/2049] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:04.381396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.391940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8112f4cc-85eb-448a-92e3-81eb476a5bd8 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.396683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5845eae-3fa4-498f-b85c-8ac8e5c0eedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.397274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5845eae-3fa4-498f-b85c-8ac8e5c0eedb admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.397561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5845eae-3fa4-498f-b85c-8ac8e5c0eedb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.397843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5845eae-3fa4-498f-b85c-8ac8e5c0eedb admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.398535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5845eae-3fa4-498f-b85c-8ac8e5c0eedb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.400123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5845eae-3fa4-498f-b85c-8ac8e5c0eedb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.410643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1037/2050] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:04.415308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.425344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ff458-27c7-499f-bb78-df89db3d21c5 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.430700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.431385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.431667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.431969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] PATH_INFO: `/v3/projects/f3df980bed574164a2066c6867c29088/users/d9f4f0e9389b4613ab3ae4e6698d7c71/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.447233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3df980bed574164a2066c6867c29088, user_id=d9f4f0e9389b4613ab3ae4e6698d7c71, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.449861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.463579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1014/2051] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:04 2022] PUT /identity/v3/projects/f3df980bed574164a2066c6867c29088/users/d9f4f0e9389b4613ab3ae4e6698d7c71/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:04.474989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.475289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.475567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.504340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] MFA Rules not processed for user `d9f4f0e9389b4613ab3ae4e6698d7c71`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:04.505530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:04.555593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:04.579954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1038/2052] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:04 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:04.594146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:04.596089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:04.609611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:04.715939 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534190f7-824b-4aa6-ac83-158425cbc383 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:04.720020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:04.720770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:04.721259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:04.721568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:04.730062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:04.731633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:04.740849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1015/2053] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:04 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:07.002087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:07.055662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8b3aed-b927-482b-827f-246f4267c610 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:07.059253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:07.059851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:07.060128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:07.060397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:07.062112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:07.124857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:07.126355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:07.135943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1039/2054] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:07.971295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:07.974283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:07.985058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79519d9-3e7b-4ddd-b6c3-7dadedfd601f service glance] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:07.989779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7fdc05-780f-45b1-9d3d-eb19d479e238 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:07.990504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7fdc05-780f-45b1-9d3d-eb19d479e238 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:07.991207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7fdc05-780f-45b1-9d3d-eb19d479e238 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:07.991207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7fdc05-780f-45b1-9d3d-eb19d479e238 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:08.057073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7fdc05-780f-45b1-9d3d-eb19d479e238 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:08.058517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7fdc05-780f-45b1-9d3d-eb19d479e238 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:08.068668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1016/2055] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:07 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:12.119314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:12.205518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882481d5-de31-43c8-862e-ecb8ea7f5f4f service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:12.211204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:12.213010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.213291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.213566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] PATH_INFO: `/v3/users/97c7dd71871f4689b01fd7433b8c8d20` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.214872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] RBAC: Authorizing `identity:delete_user(user_id=97c7dd71871f4689b01fd7433b8c8d20)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:12.217121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:12.279515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:12.285654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97c7dd71871f4689b01fd7433b8c8d20'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.293579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97c7dd71871f4689b01fd7433b8c8d20'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.307760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97c7dd71871f4689b01fd7433b8c8d20'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.309030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97c7dd71871f4689b01fd7433b8c8d20'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.318250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97c7dd71871f4689b01fd7433b8c8d20'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.329681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1040/2056] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:12 2022] DELETE /identity/v3/users/97c7dd71871f4689b01fd7433b8c8d20 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:12.331085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.331579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.331662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.353923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:12.355248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:12.445018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1017/2057] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:12.452163 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:12.453830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:12.463396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:12.465022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:12.478161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba0bda6-9c3e-4d14-871d-b50404a02c15 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:12.482961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:12.483685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.484035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.484376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] PATH_INFO: `/v3/users/861bd4018c6446a1985ad1a742262207` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.485962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=861bd4018c6446a1985ad1a742262207)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:12.487688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:12.509930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:12.530415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9a18c-7bb2-40d1-9bea-4900f0d23bec admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:12.540287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831cd83d-5a9b-456f-b165-c36d419d85fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:12.541296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cd83d-5a9b-456f-b165-c36d419d85fa service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.541296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cd83d-5a9b-456f-b165-c36d419d85fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.541543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cd83d-5a9b-456f-b165-c36d419d85fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.560386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831cd83d-5a9b-456f-b165-c36d419d85fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:12.561870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831cd83d-5a9b-456f-b165-c36d419d85fa service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:12.581741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '861bd4018c6446a1985ad1a742262207'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.583247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1041/2058] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:12.590239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '861bd4018c6446a1985ad1a742262207'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.605915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '861bd4018c6446a1985ad1a742262207'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.606819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '861bd4018c6446a1985ad1a742262207'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.615528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '861bd4018c6446a1985ad1a742262207'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:12.624758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1018/2059] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:12 2022] DELETE /identity/v3/users/861bd4018c6446a1985ad1a742262207 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:12.631295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.631295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.631295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.650615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:12.651887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:12.727269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1042/2060] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:12.752951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:12.755360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:12.847792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1608a8c2-0303-4394-bc73-e660d2bb2c7b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:12.853767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:12.854430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.855461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.856136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.873536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:12.874606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:12.921809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1019/2061] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:12.923563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:12.935025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:12.979708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4108b81-9c0b-496e-b300-9c91139d0c3d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:12.993856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:12.994509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:12.994846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:12.995063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] PATH_INFO: `/v3/projects/0d979aea8ece48538a3e6ff9bebe394c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:12.996213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d979aea8ece48538a3e6ff9bebe394c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:12.998142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:13.084041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d979aea8ece48538a3e6ff9bebe394c', 'request_id': 'req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.099659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d979aea8ece48538a3e6ff9bebe394c', 'request_id': 'req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.101695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:13.120341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d979aea8ece48538a3e6ff9bebe394c', 'request_id': 'req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.141213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] The token cache is being invalidate because project 0d979aea8ece48538a3e6ff9bebe394c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:13.141682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.142730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.145088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1043/2062] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:12 2022] DELETE /identity/v3/projects/0d979aea8ece48538a3e6ff9bebe394c => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:13.149647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:13.162361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06163731-5e73-448d-a896-b78f39cbb5c7 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:13.282106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:13.283156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.283514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.284256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] PATH_INFO: `/v3/users/7f4d578ae4384d2fbd3bd2f8689be086` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.286477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f4d578ae4384d2fbd3bd2f8689be086)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:13.296288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:13.369379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:13.372574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:13.374021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:13.384451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f4d578ae4384d2fbd3bd2f8689be086'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.396071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f4d578ae4384d2fbd3bd2f8689be086'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.405753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9af2d8-6da0-4af0-90a0-59b3e1e6b90b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:13.418303 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:13.419566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.420155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.420400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] PATH_INFO: `/v3/projects/a5c7d1ed6bc74e828c2bda14d1ab8485` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.425618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f4d578ae4384d2fbd3bd2f8689be086'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.426772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f4d578ae4384d2fbd3bd2f8689be086'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.434688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a97e5b90-8475-454d-afb8-7d3715a22b44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f4d578ae4384d2fbd3bd2f8689be086'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.441501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5c7d1ed6bc74e828c2bda14d1ab8485)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:13.442923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1020/2063] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:13 2022] DELETE /identity/v3/users/7f4d578ae4384d2fbd3bd2f8689be086 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:13.446911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:13.524160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5c7d1ed6bc74e828c2bda14d1ab8485', 'request_id': 'req-85008b45-0f6f-4eea-8f07-b37a102ff045'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.533674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5c7d1ed6bc74e828c2bda14d1ab8485', 'request_id': 'req-85008b45-0f6f-4eea-8f07-b37a102ff045'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.541517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5c7d1ed6bc74e828c2bda14d1ab8485', 'request_id': 'req-85008b45-0f6f-4eea-8f07-b37a102ff045'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.557400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] The token cache is being invalidate because project a5c7d1ed6bc74e828c2bda14d1ab8485 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:13.557704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.559544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-85008b45-0f6f-4eea-8f07-b37a102ff045 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.560452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1044/2064] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:13 2022] DELETE /identity/v3/projects/a5c7d1ed6bc74e828c2bda14d1ab8485 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:13.565644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.565931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.566232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.594834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:13.603346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:13.711130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1021/2065] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:13.719443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.719737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.721015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.741211 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:13.742257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:13.759905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1045/2066] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:13.764967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:13.780669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00751a04-9539-410d-a06b-840218f2c6e4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:13.786179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:13.787133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.787454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.787730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.788643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:13.789012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:13.790644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:13.805148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:13.805922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1022/2067] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:40:13.807226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:13.810302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:13.824294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62f07da-ae17-4eab-904d-6bfb51b03084 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:13.826003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f67a406-eb11-47aa-a9d4-e76fbcb938c1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:13.832790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:13.834416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.834416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.834848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.835458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:13.837545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:13.842476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:13.843352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:13.844330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:13.844768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] PATH_INFO: `/v3/projects/8d9082267e134b17b17641c19acbab5e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:13.846124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d9082267e134b17b17641c19acbab5e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:13.856363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:13.889575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1023/2068] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:13 2022] POST /identity/v3/projects => generated 385 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:13.894232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:13.938458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d9082267e134b17b17641c19acbab5e', 'request_id': 'req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.961785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d9082267e134b17b17641c19acbab5e', 'request_id': 'req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.972132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d9082267e134b17b17641c19acbab5e', 'request_id': 'req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.996149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] The token cache is being invalidate because project 8d9082267e134b17b17641c19acbab5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:13.996831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:13.997243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b16a95c-f76d-494c-a691-210d256c6486 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:13.998049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:14.001997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1046/2069] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:13 2022] DELETE /identity/v3/projects/8d9082267e134b17b17641c19acbab5e => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:14.006420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.008602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:14.043256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:14.088760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.089756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.090219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.090816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:14.091026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.092217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.097213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.138517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1024/2070] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:13 2022] POST /identity/v3/users => generated 293 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:14.143073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.146635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:14.156336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e77ae-4959-4d60-8fc5-f8f148c0aa95 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:14.170845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.171538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.171823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.172101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] PATH_INFO: `/v3/users/7aa8ed109a3b44ea9cc263a2c14d0d77` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.174695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7aa8ed109a3b44ea9cc263a2c14d0d77)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.176886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.178226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8c9106-d414-4506-865e-d2d5764c28ff admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:14.185407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.186425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.186860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.187702 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.188625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.191189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.206601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1025/2071] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:14 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:14.211433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.224213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124b072-e4db-480c-a598-c4e57accff24 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:14.241657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.243908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.244604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.245089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.246117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.248374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.266184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7aa8ed109a3b44ea9cc263a2c14d0d77'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:14.270542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1026/2072] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:14 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:14.272561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.283880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7aa8ed109a3b44ea9cc263a2c14d0d77'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:14.296956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87159139-c220-404a-8297-f52a13cbb96e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:14.298827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.300206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.300594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.301001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] PATH_INFO: `/v3/projects/7d04967696394240b113aa462c72aeb8/users/fca15fc3bfcb42c185a93efc9fe23de4/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.316987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7aa8ed109a3b44ea9cc263a2c14d0d77'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:14.318455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7aa8ed109a3b44ea9cc263a2c14d0d77'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:14.337923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d04967696394240b113aa462c72aeb8, user_id=fca15fc3bfcb42c185a93efc9fe23de4, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.343492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.355250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af150009-f1f1-44a3-8ba8-ca7be039aac0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7aa8ed109a3b44ea9cc263a2c14d0d77'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:14.376908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1047/2073] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:14 2022] DELETE /identity/v3/users/7aa8ed109a3b44ea9cc263a2c14d0d77 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:14.385478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1027/2074] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:14 2022] PUT /identity/v3/projects/7d04967696394240b113aa462c72aeb8/users/fca15fc3bfcb42c185a93efc9fe23de4/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:14.396477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.396773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.397055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.432736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] MFA Rules not processed for user `fca15fc3bfcb42c185a93efc9fe23de4`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:14.433977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:14.521307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1048/2075] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:14.525497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.527904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:14.619693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279ba9a5-5b18-4ec9-838e-6a6acff0e481 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:14.730785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.731458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.731790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.732205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.732873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.734893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.814708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1028/2076] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:14 2022] POST /identity/v3/projects => generated 385 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:14.816689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.833432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc9aa0-9616-4572-bfc7-9011a0852b03 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:14.940806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:14.948870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:14.949601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:14.949885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:14.950268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:14.951116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:14.953134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:14.986061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1049/2077] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:14 2022] POST /identity/v3/users => generated 293 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:14.990788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:14.998676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:15.003260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70c74c8-e1af-48da-a1b5-af82dcdbb115 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.015602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.016018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.016340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.016617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.017275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.018997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.034061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1029/2078] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:14 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:15.036142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.052449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28007ee5-16cf-4c57-8e35-bdbde7f27e27 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.060856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9b74fb-132d-4f78-8a3b-92d25ab771cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.061833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9b74fb-132d-4f78-8a3b-92d25ab771cc admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.062870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9b74fb-132d-4f78-8a3b-92d25ab771cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.063220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9b74fb-132d-4f78-8a3b-92d25ab771cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.064693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9b74fb-132d-4f78-8a3b-92d25ab771cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.066648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9b74fb-132d-4f78-8a3b-92d25ab771cc admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.078342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1050/2079] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:15 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:15.082701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.098156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276da5d9-3a15-43d5-87d8-a0f18f72c4a7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.107463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.108315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.109256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.109542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] PATH_INFO: `/v3/projects/df1c12b6bf164585a92b9e311b4ebc36/users/c7176a13a69c4ac9b4449b570cfd3b0a/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.129297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df1c12b6bf164585a92b9e311b4ebc36, user_id=c7176a13a69c4ac9b4449b570cfd3b0a, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.131896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.151005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1030/2080] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:15 2022] PUT /identity/v3/projects/df1c12b6bf164585a92b9e311b4ebc36/users/c7176a13a69c4ac9b4449b570cfd3b0a/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:15.163899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7168d337-ea20-4810-952a-024e780e5307 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.165775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7168d337-ea20-4810-952a-024e780e5307 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.166760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7168d337-ea20-4810-952a-024e780e5307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.271654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7168d337-ea20-4810-952a-024e780e5307 None None] MFA Rules not processed for user `c7176a13a69c4ac9b4449b570cfd3b0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:15.271654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7168d337-ea20-4810-952a-024e780e5307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:15.373647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.375701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:15.385287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1051/2081] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:15.392896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7168d337-ea20-4810-952a-024e780e5307 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.502407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7168d337-ea20-4810-952a-024e780e5307 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.505871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e106ab-e568-4088-a23e-9daf3368a1d9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.509284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.510031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.510496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.510957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.511699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.512162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.512875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.513231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.513587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] PATH_INFO: `/v3/projects/a6564a4b91484feb8e8cc067761a2a3d` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.514003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.514881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6564a4b91484feb8e8cc067761a2a3d)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.519424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.563222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1052/2082] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:15 2022] POST /identity/v3/projects => generated 383 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:15.567446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.585190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6564a4b91484feb8e8cc067761a2a3d', 'request_id': 'req-c537f555-4dad-4f96-9f6b-c2f99ef326bf'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:15.596559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6564a4b91484feb8e8cc067761a2a3d', 'request_id': 'req-c537f555-4dad-4f96-9f6b-c2f99ef326bf'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:15.611883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6564a4b91484feb8e8cc067761a2a3d', 'request_id': 'req-c537f555-4dad-4f96-9f6b-c2f99ef326bf'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:15.626718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] The token cache is being invalidate because project a6564a4b91484feb8e8cc067761a2a3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:15.626718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:15.626718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c537f555-4dad-4f96-9f6b-c2f99ef326bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:15.629247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1031/2083] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:15 2022] DELETE /identity/v3/projects/a6564a4b91484feb8e8cc067761a2a3d => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:15.634616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10178d83-9785-40ee-bcfb-9fec5f4272d6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.634890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10178d83-9785-40ee-bcfb-9fec5f4272d6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.635297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10178d83-9785-40ee-bcfb-9fec5f4272d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.644340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:15.674184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-10178d83-9785-40ee-bcfb-9fec5f4272d6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:15.675594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-10178d83-9785-40ee-bcfb-9fec5f4272d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:15.693778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63abcec-e22a-4f6f-93aa-0da5e4b307a2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.773355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:15.806753 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1032/2084] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:15.815048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.815730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.816186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.825490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.826234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.826589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.826977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.827792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.829977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.855146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:15.861044 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:15.868912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1053/2085] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:15 2022] POST /identity/v3/users => generated 298 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:15.869932 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.871866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:15.876119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1033/2086] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:15.880864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.886240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a92968-308f-475f-8a4d-1fcfc5f1e5bf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.894100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc082e21-06c6-4f30-a9bb-68ba0f772c6f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.897713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.898789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.899328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.899836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.901067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.902159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.902907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.903297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.903667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.904178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.906181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:15.906472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.908675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.920009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1054/2087] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:15 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:15.923930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.927625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1034/2088] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:40:15.932035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:15.945524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e195b-7efa-4d3a-8f37-1ddd71fc6659 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.955355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912802ed-18d8-4c1d-8163-1759ab5d4769 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:15.960444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3d8c0e-4fd6-413c-9007-07a6123b1325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.962516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3d8c0e-4fd6-413c-9007-07a6123b1325 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.963276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3d8c0e-4fd6-413c-9007-07a6123b1325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.963773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3d8c0e-4fd6-413c-9007-07a6123b1325 admin admin] PATH_INFO: `/v3/projects/c098173dd6de46f3a14d7259dc0388c5/users/be72616f26a54f479c8ba09f4ca3d183/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.967429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:15.968748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:15.971157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:15.971556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:15.973191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.979119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:15.984863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3d8c0e-4fd6-413c-9007-07a6123b1325 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c098173dd6de46f3a14d7259dc0388c5, user_id=be72616f26a54f479c8ba09f4ca3d183, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:15.989578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3d8c0e-4fd6-413c-9007-07a6123b1325 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.013460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1055/2089] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:15 2022] PUT /identity/v3/projects/c098173dd6de46f3a14d7259dc0388c5/users/be72616f26a54f479c8ba09f4ca3d183/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:16.013460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93c151d-ac2b-4e7c-9e20-2110035e5910 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.013460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93c151d-ac2b-4e7c-9e20-2110035e5910 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.013460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93c151d-ac2b-4e7c-9e20-2110035e5910 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.014402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1056/2090] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:40:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:40:16.015019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93c151d-ac2b-4e7c-9e20-2110035e5910 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.032238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1035/2091] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:15 2022] POST /identity/v3/projects => generated 373 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:16.050425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.122872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93c151d-ac2b-4e7c-9e20-2110035e5910 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.205507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:16.225475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cbec3b-d759-40c8-a56a-31b372bd26e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.226483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cbec3b-d759-40c8-a56a-31b372bd26e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.227012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cbec3b-d759-40c8-a56a-31b372bd26e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.227959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cbec3b-d759-40c8-a56a-31b372bd26e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.228785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cbec3b-d759-40c8-a56a-31b372bd26e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:16.230431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cbec3b-d759-40c8-a56a-31b372bd26e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.246689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1057/2092] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:16 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:16.248551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.248551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.248747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.268335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ea37df-55a1-4bb8-920e-e079fca7b4c8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.270854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:16.272347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.274653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.275435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.276998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.277932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.278715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:16.281003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.321805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1036/2093] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:16 2022] POST /identity/v3/users => generated 287 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:16.321805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.322849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.342745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4f022-eab9-4582-94a0-6db3e32f11d4 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.350883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.355929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.355929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.355929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] PATH_INFO: `/v3/users/abd58337f6804b97b1a665647b6dce3f` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.362467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] RBAC: Authorizing `identity:delete_user(user_id=abd58337f6804b97b1a665647b6dce3f)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:16.365141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.414197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1058/2094] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:40:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:16.417378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.433580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ac0e94-7ab7-4423-baf9-a385189d151b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.439516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.440201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.440513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.440789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] PATH_INFO: `/v3/projects/c098173dd6de46f3a14d7259dc0388c5/users/be72616f26a54f479c8ba09f4ca3d183/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.443609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c098173dd6de46f3a14d7259dc0388c5, user_id=be72616f26a54f479c8ba09f4ca3d183, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:16.447339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.454859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abd58337f6804b97b1a665647b6dce3f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:16.467478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abd58337f6804b97b1a665647b6dce3f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:16.471938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1059/2095] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:16 2022] PUT /identity/v3/projects/c098173dd6de46f3a14d7259dc0388c5/users/be72616f26a54f479c8ba09f4ca3d183/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:16.473679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.499864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abd58337f6804b97b1a665647b6dce3f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:16.503124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abd58337f6804b97b1a665647b6dce3f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:16.515142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d22507c0-2f32-427f-89ba-a5f4c3b5d01b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abd58337f6804b97b1a665647b6dce3f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:16.524672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1037/2096] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:16 2022] DELETE /identity/v3/users/abd58337f6804b97b1a665647b6dce3f => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:16.524672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73423da-7310-45f4-ad9a-1187f4e7418a None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.524672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73423da-7310-45f4-ad9a-1187f4e7418a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.524672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73423da-7310-45f4-ad9a-1187f4e7418a None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.533531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1038/2097] 10.222.0.20 () {54 vars in 961 bytes} [Fri Sep 9 04:40:16 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:40:16.538125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dc48e-61e4-4618-9eaf-91164cf4ecf0 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.538454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dc48e-61e4-4618-9eaf-91164cf4ecf0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.538690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dc48e-61e4-4618-9eaf-91164cf4ecf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.579170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7e6dbf-a6f2-43f4-b9cc-cee8060aef2b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.612650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ec0dc48e-61e4-4618-9eaf-91164cf4ecf0 None None] MFA Rules not processed for user `be72616f26a54f479c8ba09f4ca3d183`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:16.612650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ec0dc48e-61e4-4618-9eaf-91164cf4ecf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.631756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:16.762038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.762698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.763088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.763437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.764292 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:16.766609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.781518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1039/2098] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:16 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:16.781518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.781518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.781518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.781948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1060/2099] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:16 2022] GET /identity/v3/roles => generated 1420 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:16.788963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.790984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.812952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54493aa-9337-4a19-963b-6cd667aa46b0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.819987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.821487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.821877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.822179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] MFA Rules not processed for user `7ef39450c46b43a5bed04645689610f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:16.823438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.824172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.825262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:16.828407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:16.846523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1061/2100] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:16 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:16.847728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:16.849950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.864275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:16.960364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04eb0b1-33e8-4424-bcb3-01c54cd52026 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:16.964785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:16.966435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:16.972227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:16.973602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:16.976189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:16.982555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1040/2101] 10.222.0.20 () {58 vars in 1058 bytes} [Fri Sep 9 04:40:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:16.983464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:17.011345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a58ff0-10b0-4cd4-b202-9e1ecab0e985 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:17.019837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:17.020383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:17.020383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:17.020633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] PATH_INFO: `/v3/projects/ce9bbaae5d5e4e26a6ff2e9d4a1f98c3/users/64bf0fc10c794dc88efe25cf21ced674/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:17.052491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce9bbaae5d5e4e26a6ff2e9d4a1f98c3, user_id=64bf0fc10c794dc88efe25cf21ced674, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:17.055789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:17.074722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1041/2102] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:16 2022] PUT /identity/v3/projects/ce9bbaae5d5e4e26a6ff2e9d4a1f98c3/users/64bf0fc10c794dc88efe25cf21ced674/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:17.075933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:17.080102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:17.081633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:17.083534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:17.093772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1062/2103] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:16 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:17.096498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:17.106777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51d45a1-5e5b-498b-8964-9b5f3e60fb3b service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:17.193783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ebb407-f818-4aec-9ee7-1d1266f36a50 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:17.200528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:17.201334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:17.202623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:17.203177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] PATH_INFO: `/v3/projects/79ddbc4671b44eeba54e2eeca5387e5c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:17.204840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=79ddbc4671b44eeba54e2eeca5387e5c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:17.208865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:17.242723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:17.243390 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:17.243674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:17.243952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:17.246021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:17.263273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:17.270171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:17.274821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58e1ac2-52cb-4392-8e7a-beaab2d2ee3f service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:17.295325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1063/2104] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:17.297358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:17.297656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:17.297937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:17.309898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79ddbc4671b44eeba54e2eeca5387e5c', 'request_id': 'req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.320649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79ddbc4671b44eeba54e2eeca5387e5c', 'request_id': 'req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.328925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79ddbc4671b44eeba54e2eeca5387e5c', 'request_id': 'req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.343241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] The token cache is being invalidate because project 79ddbc4671b44eeba54e2eeca5387e5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:17.343554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.344760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.350517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1042/2105] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:17 2022] DELETE /identity/v3/projects/79ddbc4671b44eeba54e2eeca5387e5c => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:17.351570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:17.353798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:17.353946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:17.364726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:17.458411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:17.495915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1064/2106] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:17.521720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:17.523790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:17.526784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7c570c-5a78-4acd-bb35-3d03c35d71ba admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:17.545204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:17.547090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:17.547716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:17.548505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] PATH_INFO: `/v3/users/b9519a147a004ba3a30b0c5eaa65aff4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:17.551513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9519a147a004ba3a30b0c5eaa65aff4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:17.554340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:17.662310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece80dc-e0ed-4f82-8128-d31c142dfa5e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:17.685773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:17.686623 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:17.687214 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:17.687940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:17.690812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9519a147a004ba3a30b0c5eaa65aff4'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.701660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:17.707155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:17.709959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9519a147a004ba3a30b0c5eaa65aff4'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.725702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1065/2107] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:17.730040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9519a147a004ba3a30b0c5eaa65aff4'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.730908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9519a147a004ba3a30b0c5eaa65aff4'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.740719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-cb219a15-4415-449b-a9d4-e9bcc4c2c486 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9519a147a004ba3a30b0c5eaa65aff4'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:17.748493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1043/2108] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:17 2022] DELETE /identity/v3/users/b9519a147a004ba3a30b0c5eaa65aff4 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:17.961041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:17.971602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79677842-ebd5-4674-9096-03b06316989f service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.046222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.047029 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.047316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.047737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] PATH_INFO: `/v3/projects/534865bd729542edb652f63af856077e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.049023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=534865bd729542edb652f63af856077e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.052050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.119909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '534865bd729542edb652f63af856077e', 'request_id': 'req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:18.129840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '534865bd729542edb652f63af856077e', 'request_id': 'req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:18.139437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '534865bd729542edb652f63af856077e', 'request_id': 'req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:18.155796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] The token cache is being invalidate because project 534865bd729542edb652f63af856077e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:18.156143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:18.156836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-729fe98a-b1ad-4fc8-a3b6-e26b22a41e82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:18.159003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1066/2109] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:17 2022] DELETE /identity/v3/projects/534865bd729542edb652f63af856077e => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:18.163586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.163872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.164143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.199034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:18.201378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:18.242398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:18.315555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1044/2110] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:18.319525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.320073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.320231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.352193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:18.366473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:18.393331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1067/2111] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:18.398344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.413420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25622d8b-cd6b-4ac1-b6f0-f4efeb7dbced None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.419999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.420771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.421101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.421473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.422149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:18.422456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.425203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.448231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1045/2112] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:40:18.449543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.462301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fe6f21-5463-47b4-ba51-9d3835ca4582 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.467475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c1113a-7788-4c76-8dfc-230546e1ff8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.468090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c1113a-7788-4c76-8dfc-230546e1ff8c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.469360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c1113a-7788-4c76-8dfc-230546e1ff8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.469360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c1113a-7788-4c76-8dfc-230546e1ff8c admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.469360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c1113a-7788-4c76-8dfc-230546e1ff8c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.471319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c1113a-7788-4c76-8dfc-230546e1ff8c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.500973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:18.528550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1068/2113] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:18 2022] POST /identity/v3/projects => generated 385 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:18.532099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.550331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a687ca9-39f2-44ba-a2c2-c725f80c0718 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.658998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.659206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.659320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.659518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.660048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.660409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.660848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.661592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.664195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.702958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1046/2114] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:18 2022] POST /identity/v3/users => generated 293 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:18.708787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.713703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] MFA Rules not processed for user `64bf0fc10c794dc88efe25cf21ced674`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:18.715248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:18.721171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5479d16-891a-49c1-b480-0bab96fd7607 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.727452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.728148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.728509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.728849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.729571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.731663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.761607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1047/2115] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:18 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:18.766158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.781791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64303e1-d065-4b37-b947-98f9d191641d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.789011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.789722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.790068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.790407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.793808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.793808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.818972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1048/2116] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:18 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:18.822828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.834724 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cbf3ee-8998-49cc-968b-0cdd4355a033 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:18.842725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0199dbe-0704-4321-960d-c29472366b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:18.843484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0199dbe-0704-4321-960d-c29472366b8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.843732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0199dbe-0704-4321-960d-c29472366b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.844014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0199dbe-0704-4321-960d-c29472366b8a admin admin] PATH_INFO: `/v3/projects/e2ce4588581b43ab96bddf121d00135b/users/010285b680dd46069989fb6534f79e0b/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.854905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1069/2117] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:18.884364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:18.887687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0199dbe-0704-4321-960d-c29472366b8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2ce4588581b43ab96bddf121d00135b, user_id=010285b680dd46069989fb6534f79e0b, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:18.889288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:18.892242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0199dbe-0704-4321-960d-c29472366b8a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:18.927302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1049/2118] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:18 2022] PUT /identity/v3/projects/e2ce4588581b43ab96bddf121d00135b/users/010285b680dd46069989fb6534f79e0b/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:18.935020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:18.935020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:18.935020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:18.966988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] MFA Rules not processed for user `010285b680dd46069989fb6534f79e0b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:18.966988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:19.018286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e5a83-2c33-4446-be19-dfc894c34a61 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.055996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:19.095929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1050/2119] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:19.097008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.098475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:19.104607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.105416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.105843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.106264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.116394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b01ada-83ba-407d-b7e5-9807c4c0e61d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.122654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.123462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.123851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.124231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.126161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:19.187914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:19.192469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.194529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.206467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1070/2120] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:19.207438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.254904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.257121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.275358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1051/2121] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:19 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:19.307615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9251b972-9b13-4ee8-9a4c-d9a7717bf378 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.315756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.315756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.315756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.315756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.316761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.323503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.359467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1071/2122] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:19 2022] POST /identity/v3/projects => generated 385 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:19.364064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.388229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a521d614-8f86-4c73-b58f-0018351edaf4 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.477802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.478800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.479591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.481508 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.484979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.487084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.519979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1052/2123] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:19 2022] POST /identity/v3/users => generated 299 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:19.524265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.540005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc9e601-f108-4eb0-813e-4dfa98acd7cb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.550043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.551601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.552015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.552157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.554583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.559433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.590737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1072/2124] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:19.595498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.617672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47be4d49-7966-4ad6-89f5-5669d63bac7d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.628568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.629308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.629588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.629861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] PATH_INFO: `/v3/projects/f202414d9faa42f1a448a4fe767f80ec/users/cd508a4172564ee2bac3b7805258883b/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.675738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f202414d9faa42f1a448a4fe767f80ec, user_id=cd508a4172564ee2bac3b7805258883b, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.678562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.727427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1053/2125] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:19 2022] PUT /identity/v3/projects/f202414d9faa42f1a448a4fe767f80ec/users/cd508a4172564ee2bac3b7805258883b/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:19.728959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.745779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50dc16d-b373-49c0-af1f-6473dfce701e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.880964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ae5e9-deb4-4b43-8b75-9016e389a16e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.881906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ae5e9-deb4-4b43-8b75-9016e389a16e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.882451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ae5e9-deb4-4b43-8b75-9016e389a16e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.882843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ae5e9-deb4-4b43-8b75-9016e389a16e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.883744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3ae5e9-deb4-4b43-8b75-9016e389a16e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.885594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3ae5e9-deb4-4b43-8b75-9016e389a16e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.915493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1073/2126] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:19.919932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:19.932399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aedb21-b777-4550-bfba-f71c8eb9d84c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:19.946404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:19.947240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.948746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.949110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] PATH_INFO: `/v3/projects/f202414d9faa42f1a448a4fe767f80ec/users/cd508a4172564ee2bac3b7805258883b/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:19.951499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f202414d9faa42f1a448a4fe767f80ec, user_id=cd508a4172564ee2bac3b7805258883b, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:19.954300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:19.977556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1054/2127] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:19 2022] PUT /identity/v3/projects/f202414d9faa42f1a448a4fe767f80ec/users/cd508a4172564ee2bac3b7805258883b/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:19.979858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:19.980226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:19.980627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:20.049387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] MFA Rules not processed for user `cd508a4172564ee2bac3b7805258883b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:20.053445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:20.246576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:20.299347 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1074/2128] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:19 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 321 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:20.314894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:20.328931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:20.372725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:20.392711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc99675-312c-4580-a4e0-6950d51e9be0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:20.397423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c238a76-57af-4df9-8e36-b0aa8d9bec52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:20.398218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c238a76-57af-4df9-8e36-b0aa8d9bec52 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:20.398500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c238a76-57af-4df9-8e36-b0aa8d9bec52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:20.398765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c238a76-57af-4df9-8e36-b0aa8d9bec52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:20.412416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37757a87-ee44-4bbb-a1b5-3a2b85ccda7d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:20.417417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:20.418225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:20.418510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:20.418963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:20.420792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:20.487896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c238a76-57af-4df9-8e36-b0aa8d9bec52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:20.491123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c238a76-57af-4df9-8e36-b0aa8d9bec52 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:20.502150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1055/2129] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:20.503721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:20.505908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:20.518360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1075/2130] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:20 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:40:20.797302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c7d4bd-3351-4efd-a29c-778348988e63 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:20.797848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c7d4bd-3351-4efd-a29c-778348988e63 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:20.797970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c7d4bd-3351-4efd-a29c-778348988e63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:20.821980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-53c7d4bd-3351-4efd-a29c-778348988e63 None None] MFA Rules not processed for user `64bf0fc10c794dc88efe25cf21ced674`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:20.823378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-53c7d4bd-3351-4efd-a29c-778348988e63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:20.952573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1056/2131] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:24.539249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:24.540918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:24.633055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7f50ab-13fc-48a2-9251-f3ef7a66f58b service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:24.638676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:24.639482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:24.639774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:24.640065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] PATH_INFO: `/v3/users/fca15fc3bfcb42c185a93efc9fe23de4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:24.641535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] RBAC: Authorizing `identity:delete_user(user_id=fca15fc3bfcb42c185a93efc9fe23de4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:24.646862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:24.712982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:24.717460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fca15fc3bfcb42c185a93efc9fe23de4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:24.727934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fca15fc3bfcb42c185a93efc9fe23de4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:24.748839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fca15fc3bfcb42c185a93efc9fe23de4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:24.749678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fca15fc3bfcb42c185a93efc9fe23de4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:24.758839 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fca15fc3bfcb42c185a93efc9fe23de4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:24.767342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1076/2132] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:24 2022] DELETE /identity/v3/users/fca15fc3bfcb42c185a93efc9fe23de4 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:24.771694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:24.772248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:24.772248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:24.803371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:24.807613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:24.912185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1057/2133] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:24.936154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:24.938030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:25.057125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26398ae0-5f8c-4dfb-8763-b19b3525880f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:25.066276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:25.067024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:25.067507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:25.067734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:25.078194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:25.079664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:25.090325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1077/2134] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:25.306200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:25.308162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:25.382734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:25.402883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9696a3-8bbe-49c9-acbf-0a8454fc015a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:25.408744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:25.409390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:25.410032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:25.410407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:25.413200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:25.494437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:25.496836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:25.509537 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1058/2135] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:25 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:25.726487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:25.728135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:25.743952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c3b0f-9f92-4d13-b5ad-cc97a0e7fa93 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:25.764358 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:25.765505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:25.766130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:25.766912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] PATH_INFO: `/v3/projects/7d04967696394240b113aa462c72aeb8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:25.769464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d04967696394240b113aa462c72aeb8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:25.774918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:25.842862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d04967696394240b113aa462c72aeb8', 'request_id': 'req-2db848c9-945b-4cfe-a88a-e512fa1a9d95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:25.853043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d04967696394240b113aa462c72aeb8', 'request_id': 'req-2db848c9-945b-4cfe-a88a-e512fa1a9d95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:25.860962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d04967696394240b113aa462c72aeb8', 'request_id': 'req-2db848c9-945b-4cfe-a88a-e512fa1a9d95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:25.888849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] The token cache is being invalidate because project 7d04967696394240b113aa462c72aeb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:25.889451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:25.890206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:25.893447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1078/2136] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:25 2022] DELETE /identity/v3/projects/7d04967696394240b113aa462c72aeb8 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:25.897945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:25.910206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d06378-1bc6-4ef6-b713-87c7d9ce88fb service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:26.000153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:26.002225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:26.002874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:26.003641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] PATH_INFO: `/v3/users/c7176a13a69c4ac9b4449b570cfd3b0a` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:26.007161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7176a13a69c4ac9b4449b570cfd3b0a)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:26.010488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:26.087664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7176a13a69c4ac9b4449b570cfd3b0a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.098435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7176a13a69c4ac9b4449b570cfd3b0a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.114076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7176a13a69c4ac9b4449b570cfd3b0a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.114877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7176a13a69c4ac9b4449b570cfd3b0a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.122623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7176a13a69c4ac9b4449b570cfd3b0a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.128917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1059/2137] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:25 2022] DELETE /identity/v3/users/c7176a13a69c4ac9b4449b570cfd3b0a => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:26.632645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:26.635103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:26.654387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db848c9-945b-4cfe-a88a-e512fa1a9d95 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:26.706173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:26.764917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:26.765917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:26.766192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:26.766470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] PATH_INFO: `/v3/projects/df1c12b6bf164585a92b9e311b4ebc36` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:26.767657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df1c12b6bf164585a92b9e311b4ebc36)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:26.769566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:26.824283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df1c12b6bf164585a92b9e311b4ebc36', 'request_id': 'req-9b54daa8-8602-49da-b9a4-28575de29355'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.834684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df1c12b6bf164585a92b9e311b4ebc36', 'request_id': 'req-9b54daa8-8602-49da-b9a4-28575de29355'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.841475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df1c12b6bf164585a92b9e311b4ebc36', 'request_id': 'req-9b54daa8-8602-49da-b9a4-28575de29355'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.856713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] The token cache is being invalidate because project df1c12b6bf164585a92b9e311b4ebc36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:26.857011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.858013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:26.862613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1079/2138] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:26 2022] DELETE /identity/v3/projects/df1c12b6bf164585a92b9e311b4ebc36 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:26.864088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:26.865909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:26.880783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5783b137-3aee-4b50-bdad-165ff9a42cf8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:26.889114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:26.978652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:26.979337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:26.979814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:26.980098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] PATH_INFO: `/v3/users/be72616f26a54f479c8ba09f4ca3d183` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:26.981421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be72616f26a54f479c8ba09f4ca3d183)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:26.983454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:27.040243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be72616f26a54f479c8ba09f4ca3d183'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.049365 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be72616f26a54f479c8ba09f4ca3d183'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.066167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be72616f26a54f479c8ba09f4ca3d183'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.067513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be72616f26a54f479c8ba09f4ca3d183'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.074378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c5d8c7e6-8556-4604-bff1-5f3370d1e5f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be72616f26a54f479c8ba09f4ca3d183'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.083682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1060/2139] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:26 2022] DELETE /identity/v3/users/be72616f26a54f479c8ba09f4ca3d183 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:27.672309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:27.673785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:27.687729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b54daa8-8602-49da-b9a4-28575de29355 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:27.761684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:27.763659 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:27.764279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:27.764833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] PATH_INFO: `/v3/projects/c098173dd6de46f3a14d7259dc0388c5` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:27.767173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c098173dd6de46f3a14d7259dc0388c5)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:27.772759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:27.857589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c098173dd6de46f3a14d7259dc0388c5', 'request_id': 'req-8a06abcf-8479-4cc1-a5dd-3e408b890332'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.872880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c098173dd6de46f3a14d7259dc0388c5', 'request_id': 'req-8a06abcf-8479-4cc1-a5dd-3e408b890332'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.883172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c098173dd6de46f3a14d7259dc0388c5', 'request_id': 'req-8a06abcf-8479-4cc1-a5dd-3e408b890332'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.917920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] The token cache is being invalidate because project c098173dd6de46f3a14d7259dc0388c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:27.918388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.919378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8a06abcf-8479-4cc1-a5dd-3e408b890332 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:27.925244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1080/2140] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:27 2022] DELETE /identity/v3/projects/c098173dd6de46f3a14d7259dc0388c5 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:27.927821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:27.928547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:27.929021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:27.954656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:27.963135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:28.055058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1061/2141] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:28.057082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.057465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.057827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.075615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:28.113691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:28.127262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:28.137112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1081/2142] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:28.142260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.156039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d025900-2461-453a-a9d2-30b1f842aec2 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.165573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.166307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.166585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.166857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.167719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:28.168030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.169654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.183185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1062/2143] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:40:28.192988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.203060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59c37a1-1dfb-4b71-97df-280d393adfa8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.208027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.208634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.208918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.209194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.209833 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.211929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.247045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1082/2144] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:28 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:28.248136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.259095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a27429-b65d-447e-a889-2b0eb6257901 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.268727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:28.326299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.326678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.326866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.327144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.327798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.329738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.356463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1063/2145] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:28 2022] POST /identity/v3/users => generated 287 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:28.360470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.372210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ada644-06f4-468a-bf62-a0827fa858d9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.377299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.378631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.379025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.379517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.380545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.382632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.395478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1083/2146] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:28.399449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.409995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdab77b-1582-4d5a-9a14-d2fb2647b897 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.419200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b4ebb0-2ee3-43f2-b3e6-e594307da332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.419817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b4ebb0-2ee3-43f2-b3e6-e594307da332 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.420098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b4ebb0-2ee3-43f2-b3e6-e594307da332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.420369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b4ebb0-2ee3-43f2-b3e6-e594307da332 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.421021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b4ebb0-2ee3-43f2-b3e6-e594307da332 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.422842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b4ebb0-2ee3-43f2-b3e6-e594307da332 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.436918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1064/2147] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:28.442524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.453673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5354d9d1-89f1-4cab-8959-f4c871885a10 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.460580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.461325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.461610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.472337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] PATH_INFO: `/v3/projects/db42519d4833493a8ee496246978d5c2/users/c11dde0bf0704c3ba71db88360e2ffd6/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.480373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db42519d4833493a8ee496246978d5c2, user_id=c11dde0bf0704c3ba71db88360e2ffd6, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.483097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.501239 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1084/2148] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:28 2022] PUT /identity/v3/projects/db42519d4833493a8ee496246978d5c2/users/c11dde0bf0704c3ba71db88360e2ffd6/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:28.512429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.512712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.512988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.545574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] MFA Rules not processed for user `c11dde0bf0704c3ba71db88360e2ffd6`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:28.546828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:28.624219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1065/2149] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:28.637908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.640459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:28.715296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd27a8a-3a76-47ca-9630-55e86f852981 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.720181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c74558-7bde-4887-aa70-757c40333187 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.720795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c74558-7bde-4887-aa70-757c40333187 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.721387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c74558-7bde-4887-aa70-757c40333187 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.721387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c74558-7bde-4887-aa70-757c40333187 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.731314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c74558-7bde-4887-aa70-757c40333187 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.732866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c74558-7bde-4887-aa70-757c40333187 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.742977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1085/2150] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:28.771178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:28.847396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799c482-c6c4-4798-8256-1809dd5e0cb5 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:28.854406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:28.855215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:28.855608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:28.856016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] PATH_INFO: `/v3/users/c11dde0bf0704c3ba71db88360e2ffd6` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:28.857290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c11dde0bf0704c3ba71db88360e2ffd6)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:28.859747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:28.895715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:28.932735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c11dde0bf0704c3ba71db88360e2ffd6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:28.946736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c11dde0bf0704c3ba71db88360e2ffd6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:28.970555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c11dde0bf0704c3ba71db88360e2ffd6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:28.978071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c11dde0bf0704c3ba71db88360e2ffd6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:28.991431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c11dde0bf0704c3ba71db88360e2ffd6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:29.040091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1066/2151] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:28 2022] DELETE /identity/v3/users/c11dde0bf0704c3ba71db88360e2ffd6 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:29.041799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:29.042545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:29.042545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:29.107217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:29.109176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:29.223069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1086/2152] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:29.233218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:29.235540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:29.362783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e827200-6ede-4502-981b-9e58e9ce1cc0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:29.368268 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:29.368892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:29.369170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:29.369442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:29.380605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:29.382176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:29.393259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1067/2153] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:29.981174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:29.985039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:29.998666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f8d075-8613-4f69-881e-b6b08e53d417 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:30.003473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:30.005179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:30.006983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:30.007794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.008181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.008575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] PATH_INFO: `/v3/projects/db42519d4833493a8ee496246978d5c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.009826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db42519d4833493a8ee496246978d5c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:30.011917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:30.020197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad133ba-dede-45dd-bae1-37206f9e6ae5 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:30.025233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:30.025932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.026379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.026721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] PATH_INFO: `/v3/users/161e111ce6384c0caadaa06d4bcffee2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.027965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=161e111ce6384c0caadaa06d4bcffee2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:30.030018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:30.078250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:30.088439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '161e111ce6384c0caadaa06d4bcffee2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.088655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db42519d4833493a8ee496246978d5c2', 'request_id': 'req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.098672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db42519d4833493a8ee496246978d5c2', 'request_id': 'req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.102031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '161e111ce6384c0caadaa06d4bcffee2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.105523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db42519d4833493a8ee496246978d5c2', 'request_id': 'req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.117264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] The token cache is being invalidate because project db42519d4833493a8ee496246978d5c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:30.117576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.118451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.118606 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '161e111ce6384c0caadaa06d4bcffee2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.119335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '161e111ce6384c0caadaa06d4bcffee2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.120461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1087/2154] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:29 2022] DELETE /identity/v3/projects/db42519d4833493a8ee496246978d5c2 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:30.127233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4527b6ff-c932-4151-b80a-3bcfb278d0f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '161e111ce6384c0caadaa06d4bcffee2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.134886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1068/2155] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:30 2022] DELETE /identity/v3/users/161e111ce6384c0caadaa06d4bcffee2 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:40:30.368801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:30.370449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:30.451926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d64f8-26ad-4d01-8cfb-b0e521a1fdf2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:30.510234 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:30.510876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.511170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.511444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] PATH_INFO: `/v3/projects/8cc9340f30fa4e299fd096d27b9c7dbd` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.512557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cc9340f30fa4e299fd096d27b9c7dbd)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:30.514514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:30.560668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cc9340f30fa4e299fd096d27b9c7dbd', 'request_id': 'req-d7257f90-cc28-4c59-af13-9b7dcabb00b9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.568991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cc9340f30fa4e299fd096d27b9c7dbd', 'request_id': 'req-d7257f90-cc28-4c59-af13-9b7dcabb00b9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.574606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cc9340f30fa4e299fd096d27b9c7dbd', 'request_id': 'req-d7257f90-cc28-4c59-af13-9b7dcabb00b9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.586252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] The token cache is being invalidate because project 8cc9340f30fa4e299fd096d27b9c7dbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:30.586545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.587218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d7257f90-cc28-4c59-af13-9b7dcabb00b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.588712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1088/2156] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:30 2022] DELETE /identity/v3/projects/8cc9340f30fa4e299fd096d27b9c7dbd => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:30.593890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.594400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.594557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.613869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:30.615114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:30.624450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:30.683476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1069/2157] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:30.695497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.695791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.696071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.714512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:30.726404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:30.729074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:30.735219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:30.737486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:30.740189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1089/2158] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:30.744375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:30.753504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96571ebe-e873-427d-9765-d146c03a3194 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:30.759893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:30.760550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.760833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.761109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] PATH_INFO: `/v3/users/010285b680dd46069989fb6534f79e0b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.761905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f44e81-0783-4c46-87aa-dca68e0678a5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:30.762427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=010285b680dd46069989fb6534f79e0b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:30.764459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:30.770879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:30.771645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.772781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.772781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.773320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:30.773745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:30.775776 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:30.793408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1090/2159] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:40:30.797398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:30.813012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d09db9-efdb-4248-b737-672a8bb5c8a7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:30.821844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:30.822517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.822848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.823772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.824711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:30.826887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:30.836425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '010285b680dd46069989fb6534f79e0b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.849722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '010285b680dd46069989fb6534f79e0b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.871201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1091/2160] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:30 2022] POST /identity/v3/projects => generated 385 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:40:30.876112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:30.905605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '010285b680dd46069989fb6534f79e0b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.906457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '010285b680dd46069989fb6534f79e0b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.917847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ef142caf-2f12-4661-8371-20c33487fc3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '010285b680dd46069989fb6534f79e0b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:30.926896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1070/2161] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:30 2022] DELETE /identity/v3/users/010285b680dd46069989fb6534f79e0b => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:30.931319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:30.931739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:30.932054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:30.953288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:30.954868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:30.970498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ec4eb-946a-4f3e-97ce-ee5a92874640 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.019909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1071/2162] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:31.029278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.030628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:31.043345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.043965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.044254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.044518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.046295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.048265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.071906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:31.072976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1092/2163] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:30 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:31.077632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.079661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:31.093572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa543419-fe11-4eeb-8129-1ad25e2e1d48 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.100445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.101368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.101887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.102236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.103038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.104761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.106905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500a62f9-98a0-43bc-8ea1-eedefd372458 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.110696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.111342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.111625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.111902 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.115563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1093/2164] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:31.119354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.121805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.123361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.129129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da15a8e-1811-4df1-868b-0359908e63dc admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.133404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1072/2165] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:40:31.135571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50836f41-4240-48c2-ae34-c03e6d7f851a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.136192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50836f41-4240-48c2-ae34-c03e6d7f851a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.136518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50836f41-4240-48c2-ae34-c03e6d7f851a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.136886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50836f41-4240-48c2-ae34-c03e6d7f851a admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.137680 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50836f41-4240-48c2-ae34-c03e6d7f851a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.139384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50836f41-4240-48c2-ae34-c03e6d7f851a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.151277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1094/2166] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:40:31.155426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.172174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d55c2f-1dce-4c84-ac6a-495f0e8b1690 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.177386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.178034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.178308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.178579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] PATH_INFO: `/v3/projects/91bb02023bea4020ba1066ba4c85ff4d/users/2decd678421e4231bb3b58cd57d600bc/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.196761 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=91bb02023bea4020ba1066ba4c85ff4d, user_id=2decd678421e4231bb3b58cd57d600bc, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.199459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.213384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1073/2167] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:31 2022] PUT /identity/v3/projects/91bb02023bea4020ba1066ba4c85ff4d/users/2decd678421e4231bb3b58cd57d600bc/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:31.219714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.220262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.220767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.252981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:31.255626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:31.312696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:31.354492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1095/2168] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:31.368907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.429997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c63ad-0bfe-4178-8d3e-60f9bcf478ad admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.433735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.434347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.434982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.434982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.450431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.452121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.465109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1074/2169] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:31.472220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.473704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:31.487094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c426600e-7b36-498d-82b2-d170bdf5d89e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.492345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.492990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.493259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.493531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] PATH_INFO: `/v3/projects/e2ce4588581b43ab96bddf121d00135b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.495879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2ce4588581b43ab96bddf121d00135b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.498378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.560783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2ce4588581b43ab96bddf121d00135b', 'request_id': 'req-6a97989f-712f-424c-af2c-058855add616'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.572777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2ce4588581b43ab96bddf121d00135b', 'request_id': 'req-6a97989f-712f-424c-af2c-058855add616'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.581352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2ce4588581b43ab96bddf121d00135b', 'request_id': 'req-6a97989f-712f-424c-af2c-058855add616'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.594417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] The token cache is being invalidate because project e2ce4588581b43ab96bddf121d00135b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:31.594707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.595456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.597012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1096/2170] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:31 2022] DELETE /identity/v3/projects/e2ce4588581b43ab96bddf121d00135b => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:31.601572 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:31.612029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabbfd91-b658-4172-8948-6a9ba5e071d4 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:31.686439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:31.687115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:31.687396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:31.687669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] PATH_INFO: `/v3/users/cd508a4172564ee2bac3b7805258883b` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:31.688811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd508a4172564ee2bac3b7805258883b)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:31.691457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:31.749328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd508a4172564ee2bac3b7805258883b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.753035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:31.759084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd508a4172564ee2bac3b7805258883b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.777541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd508a4172564ee2bac3b7805258883b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.778410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd508a4172564ee2bac3b7805258883b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.791169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-68f93ec4-9447-4363-b122-cf967b9d1e6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd508a4172564ee2bac3b7805258883b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:31.793690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1075/2171] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:31 2022] DELETE /identity/v3/users/cd508a4172564ee2bac3b7805258883b => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:32.334008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:32.335759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:32.376734 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a97989f-712f-424c-af2c-058855add616 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:32.450651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:32.451529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:32.452525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:32.453143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] PATH_INFO: `/v3/projects/f202414d9faa42f1a448a4fe767f80ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:32.455186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f202414d9faa42f1a448a4fe767f80ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:32.457954 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:32.567606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f202414d9faa42f1a448a4fe767f80ec', 'request_id': 'req-f915ec35-25c0-4cb2-8712-a2963064aa76'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:32.580598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f202414d9faa42f1a448a4fe767f80ec', 'request_id': 'req-f915ec35-25c0-4cb2-8712-a2963064aa76'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:32.589545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f202414d9faa42f1a448a4fe767f80ec', 'request_id': 'req-f915ec35-25c0-4cb2-8712-a2963064aa76'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:32.591121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:32.610004 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] The token cache is being invalidate because project f202414d9faa42f1a448a4fe767f80ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:32.610669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:32.612188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f915ec35-25c0-4cb2-8712-a2963064aa76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:32.614575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1097/2172] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:32 2022] DELETE /identity/v3/projects/f202414d9faa42f1a448a4fe767f80ec => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:32.621217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:32.621747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:32.622138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:32.649493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:32.651340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:32.729015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1076/2173] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:32.737449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:32.738139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:32.738592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:32.762557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:32.764237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:32.774280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1098/2174] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:32.778686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:32.793418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16faf2a5-3e20-4537-9123-58793b0d5e49 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:32.801935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:32.803600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:32.804238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:32.804800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:32.805871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:32.806340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:32.808541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:32.827710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1077/2175] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:40:32.837725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:32.855298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d4f747-ee01-4cb9-8c12-fd252597c18f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:32.863309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:32.864752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:32.865174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:32.865624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:32.866550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:32.868808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:32.934499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1099/2176] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:32 2022] POST /identity/v3/projects => generated 383 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:32.938168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:32.950039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80fbc94-8aba-4abc-99f4-be8bdad0f7b4 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:33.024072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:33.024985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:33.025396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:33.026057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:33.027390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:33.030006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:33.059711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1078/2177] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:32 2022] POST /identity/v3/users => generated 292 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:33.060981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:33.075438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1af40-9d1c-4445-a21a-d2924ec277af admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:33.081360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:33.082113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:33.082506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:33.082858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:33.083626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:33.085315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:33.096213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1100/2178] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:33.100507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:33.113309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f893c8d-4ea4-41c2-aadc-58178203e3e7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:33.119707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5fad24-2225-4684-a5a0-be67976b52d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:33.120545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5fad24-2225-4684-a5a0-be67976b52d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:33.120829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5fad24-2225-4684-a5a0-be67976b52d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:33.121125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5fad24-2225-4684-a5a0-be67976b52d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:33.121787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5fad24-2225-4684-a5a0-be67976b52d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:33.123497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5fad24-2225-4684-a5a0-be67976b52d8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:33.135983 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1079/2179] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:33.140412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:33.154430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f9e8d1-22d9-4455-ac1f-f7d4c798ced8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:33.159665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:33.160422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:33.160709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:33.160982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] PATH_INFO: `/v3/projects/0a938dd67bf94816ab78222913677801/users/9f7f8f637366439d9d761139d0e77ae9/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:33.181056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a938dd67bf94816ab78222913677801, user_id=9f7f8f637366439d9d761139d0e77ae9, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:33.183893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:33.199696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1101/2180] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:33 2022] PUT /identity/v3/projects/0a938dd67bf94816ab78222913677801/users/9f7f8f637366439d9d761139d0e77ae9/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:33.211712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaa4160-7a79-4700-ad96-d7a9ee4a438b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:33.212091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaa4160-7a79-4700-ad96-d7a9ee4a438b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:33.212454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaa4160-7a79-4700-ad96-d7a9ee4a438b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:33.244511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9eaa4160-7a79-4700-ad96-d7a9ee4a438b None None] MFA Rules not processed for user `9f7f8f637366439d9d761139d0e77ae9`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:33.245763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9eaa4160-7a79-4700-ad96-d7a9ee4a438b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:33.276543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:33.333322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1080/2181] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:33.346551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:33.349687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:33.430095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d9e53a-2517-43f1-b630-3063698b5e82 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:33.434110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:33.434797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:33.435296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:33.435926 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:33.448656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:33.450539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:33.462693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1102/2182] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:36.322705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79791b4-d30c-470a-8d72-889a1ac35c5a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:36.323292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79791b4-d30c-470a-8d72-889a1ac35c5a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:36.323292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79791b4-d30c-470a-8d72-889a1ac35c5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:36.362547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f79791b4-d30c-470a-8d72-889a1ac35c5a None None] MFA Rules not processed for user `2decd678421e4231bb3b58cd57d600bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:36.364361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f79791b4-d30c-470a-8d72-889a1ac35c5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:36.433728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1081/2183] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:36.448475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:36.449939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:36.538330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7e24e-08ff-4b09-a97f-4d3f4912d0f9 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:36.543863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:36.544472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:36.544751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:36.545026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:36.555881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:36.557421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:36.567796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1103/2184] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:38.158397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71373d01-c247-4747-b9f1-b2134ea0d8b4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:38.158944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71373d01-c247-4747-b9f1-b2134ea0d8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:38.159390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71373d01-c247-4747-b9f1-b2134ea0d8b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:38.180719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-71373d01-c247-4747-b9f1-b2134ea0d8b4 None None] MFA Rules not processed for user `2decd678421e4231bb3b58cd57d600bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:38.182035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-71373d01-c247-4747-b9f1-b2134ea0d8b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:38.196558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1082/2185] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:39.368981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:39.394459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:39.475582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd2a1d0-91e5-4056-8568-c3d8157be36c service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:39.480531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:39.481543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:39.481828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:39.482101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] PATH_INFO: `/v3/users/9f7f8f637366439d9d761139d0e77ae9` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:39.483425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f7f8f637366439d9d761139d0e77ae9)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:39.485875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:39.544136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f7f8f637366439d9d761139d0e77ae9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:39.556341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f7f8f637366439d9d761139d0e77ae9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:39.570784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f7f8f637366439d9d761139d0e77ae9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:39.571547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f7f8f637366439d9d761139d0e77ae9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:39.579038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f7f8f637366439d9d761139d0e77ae9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:39.587361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1104/2186] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:39 2022] DELETE /identity/v3/users/9f7f8f637366439d9d761139d0e77ae9 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:39.592406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:39.592945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:39.592945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:39.614516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:39.616033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:39.688291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1083/2187] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:39.699740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:39.701189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:39.807831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823ccdb-6065-4349-b47e-9d4b0756d71b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:39.820482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0733005b-536c-4fee-9b38-2cbfa54ecd09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:39.821128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0733005b-536c-4fee-9b38-2cbfa54ecd09 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:39.821412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0733005b-536c-4fee-9b38-2cbfa54ecd09 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:39.821695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0733005b-536c-4fee-9b38-2cbfa54ecd09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:39.843646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0733005b-536c-4fee-9b38-2cbfa54ecd09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:39.845279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0733005b-536c-4fee-9b38-2cbfa54ecd09 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:39.860069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1105/2188] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:40.374716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.376405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:40.398031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620f3b0b-bc59-449d-bfaf-d8bb735f923e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.411557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.412218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.412501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.412845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] PATH_INFO: `/v3/projects/0a938dd67bf94816ab78222913677801` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.413946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a938dd67bf94816ab78222913677801)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.415920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.467482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:40.471497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a938dd67bf94816ab78222913677801', 'request_id': 'req-0fa28cf3-67e2-4e72-964e-1825df161ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:40.482076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a938dd67bf94816ab78222913677801', 'request_id': 'req-0fa28cf3-67e2-4e72-964e-1825df161ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:40.488998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a938dd67bf94816ab78222913677801', 'request_id': 'req-0fa28cf3-67e2-4e72-964e-1825df161ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:40.500437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] The token cache is being invalidate because project 0a938dd67bf94816ab78222913677801 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:40.500734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:40.501392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-0fa28cf3-67e2-4e72-964e-1825df161ee9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:40.502913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1084/2189] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:40 2022] DELETE /identity/v3/projects/0a938dd67bf94816ab78222913677801 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:40.507895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.508360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.508514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.529601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:40.531135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:40.566948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:40.597373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1106/2190] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:40.607590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.608066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.608171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.627957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:40.629368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:40.639311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1085/2191] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:40:40.644127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.654717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fd5a74-b079-48ca-94e7-66cb8ffc4b77 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.659397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.659996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.660909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.661185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.661845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:40:40.662154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.663840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.675799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1107/2192] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:40:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:40:40.680191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.690332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ab1aa-efe1-41bf-a95a-f877ef3277cd None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.695317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.696034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.696612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.696894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.697624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.699580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.727613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1086/2193] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:40:40 2022] POST /identity/v3/projects => generated 363 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:40.731786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.741697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa986dd-11d6-498d-9c36-f50c5eb23a75 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.796566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.797164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.797520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.797792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.798428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.800341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.821034 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1108/2194] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:40:40 2022] POST /identity/v3/users => generated 282 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:40:40.825166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.836157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56fa41-b570-4793-8a3f-fc4b6b2a8e3a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.840834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.841440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.841720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.841993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.842668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.844332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.854697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1087/2195] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:40.858648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.869057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb75f37-eecb-42ff-854b-7f8d20f7890d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.873539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ff0e4d-63f7-4cef-8b5d-684376e92d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.874123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ff0e4d-63f7-4cef-8b5d-684376e92d85 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.874401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ff0e4d-63f7-4cef-8b5d-684376e92d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.874674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ff0e4d-63f7-4cef-8b5d-684376e92d85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.875353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ff0e4d-63f7-4cef-8b5d-684376e92d85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.876930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ff0e4d-63f7-4cef-8b5d-684376e92d85 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.887816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1109/2196] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:40:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:40:40.891696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:40.902078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aa17b0-31eb-442a-aa2e-2c80a549762a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:40.906741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:40.907432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.907713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.907995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] PATH_INFO: `/v3/projects/dd5c2b59174944329169d2f5959937e1/users/95bb41f3e2c44505a2d8dd99eafef8bd/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.930899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd5c2b59174944329169d2f5959937e1, user_id=95bb41f3e2c44505a2d8dd99eafef8bd, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:40.933797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:40.950315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1088/2197] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:40:40 2022] PUT /identity/v3/projects/dd5c2b59174944329169d2f5959937e1/users/95bb41f3e2c44505a2d8dd99eafef8bd/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:40.960850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71ca78-f0c1-428e-99a8-b359761f1184 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:40.961137 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71ca78-f0c1-428e-99a8-b359761f1184 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:40.961415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71ca78-f0c1-428e-99a8-b359761f1184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:40.997583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0e71ca78-f0c1-428e-99a8-b359761f1184 None None] MFA Rules not processed for user `95bb41f3e2c44505a2d8dd99eafef8bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:40.998844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0e71ca78-f0c1-428e-99a8-b359761f1184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:41.064768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1110/2198] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:40 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:41.076639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:41.078428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:41.135849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d609abe8-9f95-4016-bb72-7af363fc64d7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:41.139797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:41.140430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:41.140708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:41.140979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:41.152150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:41.154145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:41.166387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1089/2199] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:41 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:47.670898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c980d-7b9f-414b-bea1-a8a1615a18ae None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:47.671410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c980d-7b9f-414b-bea1-a8a1615a18ae None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:47.671781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c980d-7b9f-414b-bea1-a8a1615a18ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:47.700228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4b5c980d-7b9f-414b-bea1-a8a1615a18ae None None] MFA Rules not processed for user `64bf0fc10c794dc88efe25cf21ced674`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:47.701477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4b5c980d-7b9f-414b-bea1-a8a1615a18ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:47.736977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:47.766058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1111/2200] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:50.293214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:50.296182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:50.444367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399db7c7-f8a9-481d-9948-0add049f7be1 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:50.482054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:50.482793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:50.483306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:50.483683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] PATH_INFO: `/v3/users/d9f4f0e9389b4613ab3ae4e6698d7c71` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:50.485080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9f4f0e9389b4613ab3ae4e6698d7c71)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:50.487452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:50.504464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:40:50.564614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9f4f0e9389b4613ab3ae4e6698d7c71'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:50.573968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9f4f0e9389b4613ab3ae4e6698d7c71'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:50.601630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9f4f0e9389b4613ab3ae4e6698d7c71'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:50.606336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9f4f0e9389b4613ab3ae4e6698d7c71'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:50.617031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9f4f0e9389b4613ab3ae4e6698d7c71'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:50.624327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1090/2201] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:40:50 2022] DELETE /identity/v3/users/d9f4f0e9389b4613ab3ae4e6698d7c71 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:40:50.628753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:50.629266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:50.629568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:50.656252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:40:50.657791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:50.730432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1112/2202] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:40:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:40:50.744182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:50.746458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:50.813055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9722c69e-67c7-44fd-8be4-db213c2b1cf7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:50.817763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69cc60-e10d-4907-8631-242c5b6f404e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:50.818380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69cc60-e10d-4907-8631-242c5b6f404e service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:50.818664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69cc60-e10d-4907-8631-242c5b6f404e service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:50.818955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69cc60-e10d-4907-8631-242c5b6f404e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:50.829485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69cc60-e10d-4907-8631-242c5b6f404e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:50.831133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69cc60-e10d-4907-8631-242c5b6f404e service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:50.842757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1091/2203] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:40:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:40:51.220060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:40:51.221672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:40:51.245067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed07c4-e0dc-4e48-adf3-850d8481252c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:40:51.250541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:40:51.251682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:40:51.251682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:40:51.251928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] PATH_INFO: `/v3/projects/f3df980bed574164a2066c6867c29088` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:40:51.253506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3df980bed574164a2066c6867c29088)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:40:51.256452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:40:51.306778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3df980bed574164a2066c6867c29088', 'request_id': 'req-5825dea5-4bc8-43de-b0bd-7560bbfecef6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:51.315428 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3df980bed574164a2066c6867c29088', 'request_id': 'req-5825dea5-4bc8-43de-b0bd-7560bbfecef6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:51.323840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3df980bed574164a2066c6867c29088', 'request_id': 'req-5825dea5-4bc8-43de-b0bd-7560bbfecef6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:51.335426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] The token cache is being invalidate because project f3df980bed574164a2066c6867c29088 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:40:51.335719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:51.336400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5825dea5-4bc8-43de-b0bd-7560bbfecef6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:40:51.337913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1113/2204] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:40:51 2022] DELETE /identity/v3/projects/f3df980bed574164a2066c6867c29088 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:00.837835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=48,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:01.332479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=63,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:13.098862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:13.099353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:13.099467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:13.119292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] MFA Rules not processed for user `d3c29c23d7524adf9c5a3271224ac8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:13.120500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:13.175282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1092/2205] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:13 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:16.308547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:16.309050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:16.309481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:16.338745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:16.340269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:16.422546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1114/2206] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:41:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:16.476403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:16.478533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:16.564924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52609dc-e860-498a-84a5-f99b456d57b9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:16.570425 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65203fb8-0aaa-4a0e-bfbe-9fb30fe76381 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:16.571077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65203fb8-0aaa-4a0e-bfbe-9fb30fe76381 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:16.571674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65203fb8-0aaa-4a0e-bfbe-9fb30fe76381 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:16.571951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65203fb8-0aaa-4a0e-bfbe-9fb30fe76381 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:16.580120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65203fb8-0aaa-4a0e-bfbe-9fb30fe76381 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:16.581747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65203fb8-0aaa-4a0e-bfbe-9fb30fe76381 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:16.590417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1093/2207] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:41:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:41:26.410810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=25 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:26.587563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=57 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:26.757809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:26.760094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:26.888597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13327a16-4d26-403c-86ec-f7147016c214 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:26.893210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:26.893875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:26.894951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:26.894951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] PATH_INFO: `/v3/users/95bb41f3e2c44505a2d8dd99eafef8bd` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:26.896173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=95bb41f3e2c44505a2d8dd99eafef8bd)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:26.898110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:26.979732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95bb41f3e2c44505a2d8dd99eafef8bd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:26.990217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95bb41f3e2c44505a2d8dd99eafef8bd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.009610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95bb41f3e2c44505a2d8dd99eafef8bd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.010735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95bb41f3e2c44505a2d8dd99eafef8bd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.018612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95bb41f3e2c44505a2d8dd99eafef8bd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.029055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1115/2208] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:41:26 2022] DELETE /identity/v3/users/95bb41f3e2c44505a2d8dd99eafef8bd => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:27.030807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:27.031230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:27.031602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:27.058256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:27.059974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:27.148789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1094/2209] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:27.171681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:27.173278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:27.241415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b00ccd9-372a-476f-81fc-04c4932ccf6f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:27.245477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423b4887-ab1d-43fa-8cdd-b240750e3c29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:27.246279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b4887-ab1d-43fa-8cdd-b240750e3c29 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:27.246709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b4887-ab1d-43fa-8cdd-b240750e3c29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:27.247195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b4887-ab1d-43fa-8cdd-b240750e3c29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:27.258109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423b4887-ab1d-43fa-8cdd-b240750e3c29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:27.259856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423b4887-ab1d-43fa-8cdd-b240750e3c29 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:27.275119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1116/2210] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:41:27.765771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:27.768021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:27.786892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfeb302-7208-4b48-9cb2-a40b80261c75 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:27.798817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:27.799530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:27.799827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:27.800360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] PATH_INFO: `/v3/projects/dd5c2b59174944329169d2f5959937e1` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:27.801483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd5c2b59174944329169d2f5959937e1)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:27.804660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:27.876681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd5c2b59174944329169d2f5959937e1', 'request_id': 'req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.904190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd5c2b59174944329169d2f5959937e1', 'request_id': 'req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.921174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd5c2b59174944329169d2f5959937e1', 'request_id': 'req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.938984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] The token cache is being invalidate because project dd5c2b59174944329169d2f5959937e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:41:27.939289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.940206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ce7cb29d-1bc8-4fe3-9a85-7afbe8f10718 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:27.942969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1095/2211] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:41:27 2022] DELETE /identity/v3/projects/dd5c2b59174944329169d2f5959937e1 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:27.947817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:27.948327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:27.948421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:27.975556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:27.976789 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:28.001419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:28.057714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1117/2212] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:28.067217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.067625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.067999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.118969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:28.120241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:28.148395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1096/2213] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:28.154018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.166669 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9791f3fe-10f2-42a2-89e9-27683e293e00 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:28.179729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:28.182077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.182361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.182634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.183319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:41:28.184959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:28.187810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:28.222823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1118/2214] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:41:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:41:28.228305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.249901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6123b34-9c53-4031-810d-41010bb1bd09 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:28.277271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:28.277917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.278671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.278995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.279660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:28.282624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:28.333564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1097/2215] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:41:28 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:41:28.337614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.350739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f14295c-2279-49bb-aa9f-11ceb9347993 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:28.454756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:28.455443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.455739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.456531 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.457212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:28.459761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:28.511133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1119/2216] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:41:28 2022] POST /identity/v3/users => generated 287 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:41:28.512086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.525813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711b420-103b-4960-aef6-578ad1736234 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:28.540929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:28.542137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.542633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.543179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.544183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:28.547105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:28.569116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.582680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1098/2217] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:41:28.584945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15271ec4-533d-4501-9f48-99da3ef11979 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:28.594780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b443ab-12cf-4d6c-a877-0d9b803934f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:28.595561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b443ab-12cf-4d6c-a877-0d9b803934f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.595928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b443ab-12cf-4d6c-a877-0d9b803934f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.597875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b443ab-12cf-4d6c-a877-0d9b803934f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.598815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b443ab-12cf-4d6c-a877-0d9b803934f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:28.607909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b443ab-12cf-4d6c-a877-0d9b803934f5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:28.625328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1120/2218] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:41:28.629473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.645797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f81e5bc-4039-4a0a-b9e5-38cf15b01ccd admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:28.653842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:28.654742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.655146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.655505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] PATH_INFO: `/v3/projects/4eae27a37d794c4ea3210540bf48e096/users/292619b506cc4920ab32c71e77810b05/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.677644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4eae27a37d794c4ea3210540bf48e096, user_id=292619b506cc4920ab32c71e77810b05, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:28.680842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:28.710284 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1099/2219] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:41:28 2022] PUT /identity/v3/projects/4eae27a37d794c4ea3210540bf48e096/users/292619b506cc4920ab32c71e77810b05/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:28.731882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30439819-3f8e-423b-bd43-2131b28c989e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:28.732561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30439819-3f8e-423b-bd43-2131b28c989e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:28.733335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30439819-3f8e-423b-bd43-2131b28c989e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:28.778455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-30439819-3f8e-423b-bd43-2131b28c989e None None] MFA Rules not processed for user `292619b506cc4920ab32c71e77810b05`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:28.795817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-30439819-3f8e-423b-bd43-2131b28c989e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:28.909795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1121/2220] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:41:28.923597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:28.925955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:28.959671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:29.021681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66b7ef-1cc5-44ae-b419-d70262cd5e65 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:29.026196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:29.027035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:29.027485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:29.027758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:29.037450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:29.039274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:29.051200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1100/2221] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:41:36.107339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:36.107845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:36.107940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:36.127293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:36.128632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:36.141089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:36.190655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1122/2222] 10.222.0.20 () {58 vars in 967 bytes} [Fri Sep 9 04:41:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:36.203540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:36.205015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:36.276254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f3861-7e14-45ff-bfea-57cbadd03369 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:36.281265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:36.281937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:36.282213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:36.282488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:36.291795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:36.293406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:36.303852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1101/2223] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:41:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:41:38.457079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:38.458580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:38.523987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03e56c7-fdc4-4c85-973b-6ad0cff14fba None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:38.528538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:38.529162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:38.529447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:38.529724 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] PATH_INFO: `/v3/users/64bf0fc10c794dc88efe25cf21ced674` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:38.530867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] RBAC: Authorizing `identity:delete_user(user_id=64bf0fc10c794dc88efe25cf21ced674)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:38.532818 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:38.577929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64bf0fc10c794dc88efe25cf21ced674'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.584574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64bf0fc10c794dc88efe25cf21ced674'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.597530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64bf0fc10c794dc88efe25cf21ced674'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.598203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64bf0fc10c794dc88efe25cf21ced674'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.605794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64bf0fc10c794dc88efe25cf21ced674'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.611782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1123/2224] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:41:38 2022] DELETE /identity/v3/users/64bf0fc10c794dc88efe25cf21ced674 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:38.793914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:38.804838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5dcd82-d024-42f5-9b18-033c506b7d16 service swift] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:38.862464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:38.863071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:38.863352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:38.863659 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] PATH_INFO: `/v3/projects/ce9bbaae5d5e4e26a6ff2e9d4a1f98c3` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:38.864791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce9bbaae5d5e4e26a6ff2e9d4a1f98c3)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:38.866705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:38.916312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce9bbaae5d5e4e26a6ff2e9d4a1f98c3', 'request_id': 'req-893944ad-1a71-48b3-9867-647b8d34e319'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.920459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:38.924595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce9bbaae5d5e4e26a6ff2e9d4a1f98c3', 'request_id': 'req-893944ad-1a71-48b3-9867-647b8d34e319'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.932129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce9bbaae5d5e4e26a6ff2e9d4a1f98c3', 'request_id': 'req-893944ad-1a71-48b3-9867-647b8d34e319'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.943800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] The token cache is being invalidate because project ce9bbaae5d5e4e26a6ff2e9d4a1f98c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:41:38.944090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.944731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-893944ad-1a71-48b3-9867-647b8d34e319 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:38.946233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1102/2225] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:41:38 2022] DELETE /identity/v3/projects/ce9bbaae5d5e4e26a6ff2e9d4a1f98c3 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:39.810824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:39.812271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:39.825812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5933cd-c492-4f97-a70e-757c2e8f0503 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:39.864578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:39.883681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:39.884314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:39.884598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:39.884876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] PATH_INFO: `/v3/users/292619b506cc4920ab32c71e77810b05` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:39.886007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] RBAC: Authorizing `identity:delete_user(user_id=292619b506cc4920ab32c71e77810b05)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:39.887960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:39.934384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '292619b506cc4920ab32c71e77810b05'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:39.942673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '292619b506cc4920ab32c71e77810b05'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:39.956310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '292619b506cc4920ab32c71e77810b05'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:39.957466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '292619b506cc4920ab32c71e77810b05'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:39.964702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '292619b506cc4920ab32c71e77810b05'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:39.971247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1124/2226] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:41:39 2022] DELETE /identity/v3/users/292619b506cc4920ab32c71e77810b05 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:39.975821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:39.976273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:39.976401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:39.996982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:39.998207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:40.057741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1103/2227] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:40.067804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.069260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:40.126642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4c6e12-7f4b-411f-b896-36efa6d13076 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.130374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb3615a-1802-4619-9b4a-c8b25ca4607d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.131019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb3615a-1802-4619-9b4a-c8b25ca4607d service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.131307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb3615a-1802-4619-9b4a-c8b25ca4607d service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.131667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb3615a-1802-4619-9b4a-c8b25ca4607d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.140655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb3615a-1802-4619-9b4a-c8b25ca4607d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.142081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb3615a-1802-4619-9b4a-c8b25ca4607d service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:40.151443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1125/2228] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:41:40.432642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.434090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:40.452701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9623a28c-1bd8-4fe1-b523-df636270d669 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.459448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.460720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.461662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.463135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] PATH_INFO: `/v3/projects/4eae27a37d794c4ea3210540bf48e096` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.464880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4eae27a37d794c4ea3210540bf48e096)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.467128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:40.517531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4eae27a37d794c4ea3210540bf48e096', 'request_id': 'req-2566335b-a7c8-486a-a2ba-ed4e5a13529d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:40.527135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4eae27a37d794c4ea3210540bf48e096', 'request_id': 'req-2566335b-a7c8-486a-a2ba-ed4e5a13529d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:40.533532 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4eae27a37d794c4ea3210540bf48e096', 'request_id': 'req-2566335b-a7c8-486a-a2ba-ed4e5a13529d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:40.545726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] The token cache is being invalidate because project 4eae27a37d794c4ea3210540bf48e096 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:41:40.546026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:40.546704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2566335b-a7c8-486a-a2ba-ed4e5a13529d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:41:40.548449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1104/2229] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:41:40 2022] DELETE /identity/v3/projects/4eae27a37d794c4ea3210540bf48e096 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:40.553230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.553523 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.553806 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.577867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:40.579782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:40.662766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1126/2230] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:40.671605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.672055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.672473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.693177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:40.694464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:40.703068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1105/2231] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:40.708794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.716296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:40.720475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c821c92d-c4b9-4bb9-82b4-a6c3b96c81bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.730577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.731354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.731689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.732572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.733226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:41:40.733538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.735304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:40.751884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1127/2232] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:41:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:41:40.753158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.763691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc90c9a-5f37-4206-be26-e3b6be732fc6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.769335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.769960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.770245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.770524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.771259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.773411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:40.813070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1106/2233] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:41:40 2022] POST /identity/v3/projects => generated 389 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:41:40.817176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.828313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82958b2-68b1-4e1f-a880-9889f13caab4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.897464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.898081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.898365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.899299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.899973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.902612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:40.928974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1128/2234] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:41:40 2022] POST /identity/v3/users => generated 295 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:41:40.930363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.941656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a22fc6-b321-420c-a03d-cb28d93196b9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.947492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.948176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.948522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.948914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.949704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.951427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:40.966993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1107/2235] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:41:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:41:40.969423 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:40.981016 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91a0baf-93db-4d1e-bfd4-901dc9ae2133 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:40.986815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233dd0d-2f02-4d57-a98c-9d545c4b7072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:40.987920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233dd0d-2f02-4d57-a98c-9d545c4b7072 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:40.988342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233dd0d-2f02-4d57-a98c-9d545c4b7072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:40.988773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233dd0d-2f02-4d57-a98c-9d545c4b7072 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:40.989572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5233dd0d-2f02-4d57-a98c-9d545c4b7072 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:40.991438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5233dd0d-2f02-4d57-a98c-9d545c4b7072 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:41.004123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1129/2236] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:41:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:41:41.009620 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:41.022601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b65812c-b6d1-4e0a-8bb7-b5e20e2ed696 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:41.028061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:41.028911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:41.029358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:41.029636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] PATH_INFO: `/v3/projects/9e9febf4712940189ade2cc278c02b36/users/c0960afe06684c47b9ccdeed43a18d79/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:41.045317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e9febf4712940189ade2cc278c02b36, user_id=c0960afe06684c47b9ccdeed43a18d79, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:41.048222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:41.065442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1108/2237] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:41:41 2022] PUT /identity/v3/projects/9e9febf4712940189ade2cc278c02b36/users/c0960afe06684c47b9ccdeed43a18d79/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:41:41.076496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415455f-e04d-4f51-b370-f08dcb80a287 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:41.077612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415455f-e04d-4f51-b370-f08dcb80a287 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:41.077961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415455f-e04d-4f51-b370-f08dcb80a287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:41.108916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d415455f-e04d-4f51-b370-f08dcb80a287 None None] MFA Rules not processed for user `c0960afe06684c47b9ccdeed43a18d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:41.110250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d415455f-e04d-4f51-b370-f08dcb80a287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:41.177831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1130/2238] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:41:41 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:41.189475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:41.191526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:41.197898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:41.252708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7b2c5-9cde-4d11-8757-9c9b809b6e28 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:41.256465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:41.257081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:41.257366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:41.257647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:41.267521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:41.269058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:41.277932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1109/2239] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:41 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:41:42.805894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59de0be-313c-4674-b342-ab2cedc49904 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:42.806448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59de0be-313c-4674-b342-ab2cedc49904 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:42.806448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59de0be-313c-4674-b342-ab2cedc49904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:42.829074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d59de0be-313c-4674-b342-ab2cedc49904 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:42.830304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d59de0be-313c-4674-b342-ab2cedc49904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:42.897222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1131/2240] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:41:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:42.910159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:42.911730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:42.982793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b2c3c2-a464-4c20-82f5-76684cbf861f service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:42.987943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e15b9c-a879-4d2b-82c8-e2de6bf78839 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:42.988544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e15b9c-a879-4d2b-82c8-e2de6bf78839 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:42.988819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e15b9c-a879-4d2b-82c8-e2de6bf78839 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:42.989092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e15b9c-a879-4d2b-82c8-e2de6bf78839 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:42.997207 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e15b9c-a879-4d2b-82c8-e2de6bf78839 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:42.998784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e15b9c-a879-4d2b-82c8-e2de6bf78839 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:43.006814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1110/2241] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:41:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:41:52.890963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:53.003561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=57 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:41:59.142590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a635be8-2b8a-4626-b73a-16b0d81c65f6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.143165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a635be8-2b8a-4626-b73a-16b0d81c65f6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.143289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a635be8-2b8a-4626-b73a-16b0d81c65f6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.144586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1132/2242] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:41:59.151234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b83f27-4c2a-4811-b886-7d4fcc70d5dc None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.151703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b83f27-4c2a-4811-b886-7d4fcc70d5dc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.151816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b83f27-4c2a-4811-b886-7d4fcc70d5dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.171916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-97b83f27-4c2a-4811-b886-7d4fcc70d5dc None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:59.173169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-97b83f27-4c2a-4811-b886-7d4fcc70d5dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:59.248812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1111/2243] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:41:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:59.260576 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a635be8-2b8a-4626-b73a-16b0d81c65f6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:59.262687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6a635be8-2b8a-4626-b73a-16b0d81c65f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:59.321242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a635be8-2b8a-4626-b73a-16b0d81c65f6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:59.324807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d65921-14e7-43f6-946b-374108de9ba5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:59.325405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d65921-14e7-43f6-946b-374108de9ba5 service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.325685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d65921-14e7-43f6-946b-374108de9ba5 service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.325960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d65921-14e7-43f6-946b-374108de9ba5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.336021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d65921-14e7-43f6-946b-374108de9ba5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:59.338007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d65921-14e7-43f6-946b-374108de9ba5 service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:59.349024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1133/2244] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:41:59.418264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaabd24f-0024-449c-bf13-ae8ccdbbc0d0 None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.418557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaabd24f-0024-449c-bf13-ae8ccdbbc0d0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.418841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaabd24f-0024-449c-bf13-ae8ccdbbc0d0 None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.420241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1112/2245] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:41:59.425194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321821c4-31ec-4ce5-9c11-1840d0203553 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.425429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321821c4-31ec-4ce5-9c11-1840d0203553 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.425732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321821c4-31ec-4ce5-9c11-1840d0203553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.445447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-321821c4-31ec-4ce5-9c11-1840d0203553 None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:59.446646 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-321821c4-31ec-4ce5-9c11-1840d0203553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:59.496817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1134/2246] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:41:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:59.509352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaabd24f-0024-449c-bf13-ae8ccdbbc0d0 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:59.511102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-eaabd24f-0024-449c-bf13-ae8ccdbbc0d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:59.524011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaabd24f-0024-449c-bf13-ae8ccdbbc0d0 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:59.528426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c4ae29-5fab-4a3c-b431-6a49cc628840 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:59.529021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c4ae29-5fab-4a3c-b431-6a49cc628840 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.529300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c4ae29-5fab-4a3c-b431-6a49cc628840 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.529598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c4ae29-5fab-4a3c-b431-6a49cc628840 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.536546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c4ae29-5fab-4a3c-b431-6a49cc628840 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:59.538062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c4ae29-5fab-4a3c-b431-6a49cc628840 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:59.546226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1113/2247] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:41:59.628065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c019da-f38f-4160-858f-d153f96b7498 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.628361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c019da-f38f-4160-858f-d153f96b7498 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.628644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c019da-f38f-4160-858f-d153f96b7498 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.630022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1135/2248] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:41:59.636291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.636578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.636857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.657595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:41:59.658801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:59.668437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1114/2249] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:41:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:41:59.678890 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c019da-f38f-4160-858f-d153f96b7498 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:41:59.680745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-78c019da-f38f-4160-858f-d153f96b7498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:41:59.697138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c019da-f38f-4160-858f-d153f96b7498 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:41:59.701192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa346ec-07bf-4f13-8b61-9f72b4e39b1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:41:59.701794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa346ec-07bf-4f13-8b61-9f72b4e39b1b service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:41:59.702075 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa346ec-07bf-4f13-8b61-9f72b4e39b1b service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:41:59.702381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa346ec-07bf-4f13-8b61-9f72b4e39b1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:41:59.713910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa346ec-07bf-4f13-8b61-9f72b4e39b1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:41:59.715363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa346ec-07bf-4f13-8b61-9f72b4e39b1b service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:41:59.726516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1136/2250] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:41:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:42:00.787629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:42:00.852964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5e3b7-5545-484a-b443-fdc4367e3e1e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:42:00.858095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:42:00.858730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:00.859049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:00.859326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] PATH_INFO: `/v3/users/c0960afe06684c47b9ccdeed43a18d79` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:00.860462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0960afe06684c47b9ccdeed43a18d79)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:42:00.862383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:42:00.910492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0960afe06684c47b9ccdeed43a18d79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:00.919824 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0960afe06684c47b9ccdeed43a18d79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:00.932555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0960afe06684c47b9ccdeed43a18d79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:00.933391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0960afe06684c47b9ccdeed43a18d79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:00.935338 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:00.941624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0960afe06684c47b9ccdeed43a18d79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:00.949782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1115/2251] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:42:00 2022] DELETE /identity/v3/users/c0960afe06684c47b9ccdeed43a18d79 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:42:00.954230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:00.954727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:00.954813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:00.974736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:42:00.975991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:01.036455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1137/2252] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:42:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:42:01.046036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:42:01.047558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:01.106901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c4a4b-f22a-47c2-99a5-76f3058bda2d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:42:01.110465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:42:01.111107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:01.111387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:01.111658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:01.120864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:42:01.122304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:42:01.132094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1116/2253] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:42:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:42:01.415682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:42:01.417685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:01.430174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556306f9-ba8e-42fd-960f-bec7040cb156 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:42:01.435634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:42:01.436277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:01.436557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:01.436836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] PATH_INFO: `/v3/projects/9e9febf4712940189ade2cc278c02b36` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:01.439031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e9febf4712940189ade2cc278c02b36)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:42:01.441551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:42:01.481263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:01.491424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e9febf4712940189ade2cc278c02b36', 'request_id': 'req-0337794b-f5f9-4c17-b786-033f6901e2f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:01.499920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e9febf4712940189ade2cc278c02b36', 'request_id': 'req-0337794b-f5f9-4c17-b786-033f6901e2f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:01.505794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e9febf4712940189ade2cc278c02b36', 'request_id': 'req-0337794b-f5f9-4c17-b786-033f6901e2f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:01.517788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] The token cache is being invalidate because project 9e9febf4712940189ade2cc278c02b36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:42:01.518079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:01.518725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0337794b-f5f9-4c17-b786-033f6901e2f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:42:01.520235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1138/2254] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:42:01 2022] DELETE /identity/v3/projects/9e9febf4712940189ade2cc278c02b36 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:42:05.704759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:42:05.706337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:05.779873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58711f9-43ee-4a5c-83f1-403b004df432 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:42:05.844608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:42:05.845559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:05.845940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:05.846311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:05.848140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:05.859573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:42:05.861258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b1b16d-82ba-472b-a972-7897beed8dd7 service placement] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:42:05.871991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1117/2255] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:42:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:42:07.294854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:07.295408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:07.295579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:07.315296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] MFA Rules not processed for user `2decd678421e4231bb3b58cd57d600bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:42:07.316661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:07.373832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1139/2256] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:42:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:42:08.186748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b7e16-8857-4800-8def-936ddbe348ea None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:08.187294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b7e16-8857-4800-8def-936ddbe348ea None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:08.187411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b7e16-8857-4800-8def-936ddbe348ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:08.210306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-df7b7e16-8857-4800-8def-936ddbe348ea None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:42:08.210306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-df7b7e16-8857-4800-8def-936ddbe348ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:08.231769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:08.293241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1118/2257] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:42:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:42:08.310099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:42:08.311731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:08.382847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ddbf47-df1a-44f3-b127-d0c2f9606ca8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:42:08.387862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:42:08.388472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:08.388754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:08.389035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:08.397316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:42:08.398913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:42:08.408420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1140/2258] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:42:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:42:18.274067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=13 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:18.404905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=5,keystone:SELECT=68,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:30.628876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b5cf9-b5e2-4aa3-a105-a488c0bb5b83 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:30.629385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b5cf9-b5e2-4aa3-a105-a488c0bb5b83 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:30.629500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b5cf9-b5e2-4aa3-a105-a488c0bb5b83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:30.649153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-053b5cf9-b5e2-4aa3-a105-a488c0bb5b83 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:42:30.650352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-053b5cf9-b5e2-4aa3-a105-a488c0bb5b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:30.657545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1119/2259] 10.222.0.20 () {58 vars in 967 bytes} [Fri Sep 9 04:42:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:42:30.680394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:42:30.691810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f414ac28-b79e-41fe-ad6b-d1a85243df16 service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:42:30.697555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bead8-3da5-4289-bda4-fe8c4e8d538c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:42:30.698731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bead8-3da5-4289-bda4-fe8c4e8d538c service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:30.699213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bead8-3da5-4289-bda4-fe8c4e8d538c service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:30.699498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bead8-3da5-4289-bda4-fe8c4e8d538c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:30.708529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bead8-3da5-4289-bda4-fe8c4e8d538c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:42:30.710205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bead8-3da5-4289-bda4-fe8c4e8d538c service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:42:30.718316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1141/2260] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:42:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:42:40.643666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:40.715062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:42:51.775757 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89704161-1a87-44f8-9c98-4553e861887f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:42:51.776316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89704161-1a87-44f8-9c98-4553e861887f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:42:51.776316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89704161-1a87-44f8-9c98-4553e861887f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:42:51.795176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-89704161-1a87-44f8-9c98-4553e861887f None None] MFA Rules not processed for user `d3c29c23d7524adf9c5a3271224ac8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:42:51.796370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-89704161-1a87-44f8-9c98-4553e861887f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:42:51.850164 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1120/2261] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:42:51 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:43:01.843691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=25 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:43:20.207050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe560c9-5673-4227-a5ab-970817a5cd40 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:43:20.207589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe560c9-5673-4227-a5ab-970817a5cd40 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:43:20.207589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe560c9-5673-4227-a5ab-970817a5cd40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:43:20.226688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dfe560c9-5673-4227-a5ab-970817a5cd40 None None] MFA Rules not processed for user `befee01f638c4e7681b30c64acad6a6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:43:20.227930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dfe560c9-5673-4227-a5ab-970817a5cd40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:20.289418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1142/2262] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:43:20 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:43:28.837784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:43:28.838302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:43:28.838402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:43:28.857683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] MFA Rules not processed for user `bf68e214fd4740be96f2d6c4325f6ee9`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:43:28.858882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:28.911582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1121/2263] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:43:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:43:30.280088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=25 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:43:34.958004 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:43:34.958612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:43:34.958612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:43:34.977766 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] MFA Rules not processed for user `2decd678421e4231bb3b58cd57d600bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:43:34.978997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:34.985676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1143/2264] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:43:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:43:38.904940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=25 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:43:44.972604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:43:56.177691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:43:56.179133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:56.234185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ce792-f776-4411-a98b-90a7ed07ad34 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:43:56.237750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:43:56.238359 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:43:56.238636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:43:56.238904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:43:56.240650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:56.311614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:43:56.312972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:43:56.324172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1122/2265] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:43:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:43:57.518737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:43:57.529198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fefb834-329f-4bd3-830c-38c4a22867aa None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:43:57.533011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:43:57.533618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:43:57.533900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:43:57.534178 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:43:57.535981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:57.551501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:43:57.552943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:43:57.564965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1144/2266] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:43:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:43:58.678099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:43:58.686802 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56c7f47-22dd-4419-b3b1-58d827e752ac service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:43:58.690405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:43:58.691278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:43:58.691562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:43:58.691836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:43:58.693574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:43:58.755234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:43:58.756711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:43:58.766947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1123/2267] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:43:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:01.689746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:01.691510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:01.705461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c6db89-cd3f-4428-93a4-95edd6bed168 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:01.709911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:01.710556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:01.711042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:01.711315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] PATH_INFO: `/v3/users/d3c29c23d7524adf9c5a3271224ac8d3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:01.712481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3c29c23d7524adf9c5a3271224ac8d3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:01.714379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:01.761324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3c29c23d7524adf9c5a3271224ac8d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:01.772142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3c29c23d7524adf9c5a3271224ac8d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:01.785465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3c29c23d7524adf9c5a3271224ac8d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:01.786144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3c29c23d7524adf9c5a3271224ac8d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:01.793608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2b0575cc-122f-4cd2-a3d4-b4071eb5ee58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3c29c23d7524adf9c5a3271224ac8d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:01.799521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1145/2268] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:01 2022] DELETE /identity/v3/users/d3c29c23d7524adf9c5a3271224ac8d3 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:01.980143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:01.989936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd06d686-531b-4c9f-b291-d2fd70e8f213 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.024250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:02.050015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.050656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.050963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.051232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] PATH_INFO: `/v3/projects/1d203c97242044fb8bf29bdc461e2efd` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.052376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d203c97242044fb8bf29bdc461e2efd)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.054302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.098691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d203c97242044fb8bf29bdc461e2efd', 'request_id': 'req-e927a354-3e2b-4c28-94d3-099fe26fdc87'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:02.106161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d203c97242044fb8bf29bdc461e2efd', 'request_id': 'req-e927a354-3e2b-4c28-94d3-099fe26fdc87'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:02.111683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d203c97242044fb8bf29bdc461e2efd', 'request_id': 'req-e927a354-3e2b-4c28-94d3-099fe26fdc87'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:02.122448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] The token cache is being invalidate because project 1d203c97242044fb8bf29bdc461e2efd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:02.122737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:02.123434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e927a354-3e2b-4c28-94d3-099fe26fdc87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:02.124956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1124/2269] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:01 2022] DELETE /identity/v3/projects/1d203c97242044fb8bf29bdc461e2efd => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:02.129857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.130144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.130420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.149655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:02.151412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:02.209699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1146/2270] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:02.218011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.218300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.218578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.238897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:02.240140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:02.248078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1125/2271] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:02.252242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.261997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef29fd9-b2a6-44ca-ad41-9c0f531c442c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.266573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.267210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.267487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.267758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.268406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:44:02.268710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.270295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.282335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1147/2272] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:44:02.286721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.296701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0771a144-57f4-4476-ac5d-c1fbab530a04 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.301336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.301925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.302200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.302467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.303190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.305070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.332046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1126/2273] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:02 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:02.336264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.346571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c937d-d150-4a6c-ad26-7608107649c1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.389525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:02.403330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.403862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.404140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.404411 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.405035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.406924 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.426968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1148/2274] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:02 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:02.431049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.442220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d18f56-0ac7-4a26-a12c-461e730396f5 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.446865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.447481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.447758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.448028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.448667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.450247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.459990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1127/2275] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:02.463991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.473870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522d3ab-58d4-4141-94e6-dde774281ef3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.479032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8363509-ed7e-40a2-ad9f-c6b4793daad5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.479622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8363509-ed7e-40a2-ad9f-c6b4793daad5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.479905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8363509-ed7e-40a2-ad9f-c6b4793daad5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.480184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8363509-ed7e-40a2-ad9f-c6b4793daad5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.480826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8363509-ed7e-40a2-ad9f-c6b4793daad5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.482409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8363509-ed7e-40a2-ad9f-c6b4793daad5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.492836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1149/2276] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:02.497373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.507120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b57c2-6392-4a01-9448-c3a8656860f5 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.512045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.512678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.512960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.513274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] PATH_INFO: `/v3/projects/b3b2f87c566844beb02d98fa01082bfe/users/a7908f2dc30a4961a4b50a40f5c706d0/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.528878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3b2f87c566844beb02d98fa01082bfe, user_id=a7908f2dc30a4961a4b50a40f5c706d0, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.531585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.545647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1128/2277] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:02 2022] PUT /identity/v3/projects/b3b2f87c566844beb02d98fa01082bfe/users/a7908f2dc30a4961a4b50a40f5c706d0/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:02.557284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.557588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.557865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.586235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] MFA Rules not processed for user `a7908f2dc30a4961a4b50a40f5c706d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:02.587458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:02.651283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1150/2278] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:02.662504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:02.664174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:02.724613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:02.741735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b087a0-9e9f-4007-88d3-8de24f251d0e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:02.747906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfd89b1-0648-466e-b09c-2f19980ab7ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:02.748510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd89b1-0648-466e-b09c-2f19980ab7ad service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:02.748822 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd89b1-0648-466e-b09c-2f19980ab7ad service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:02.749092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd89b1-0648-466e-b09c-2f19980ab7ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:02.756760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfd89b1-0648-466e-b09c-2f19980ab7ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:02.758339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfd89b1-0648-466e-b09c-2f19980ab7ad service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:02.766839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1129/2279] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:02 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:05.445702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:05.504647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbf8ae2-64af-43ec-9f20-9b92074c5794 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:05.509903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:05.510527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:05.510809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:05.511170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] PATH_INFO: `/v3/users/a7908f2dc30a4961a4b50a40f5c706d0` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:05.512293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7908f2dc30a4961a4b50a40f5c706d0)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:05.514419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:05.559803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7908f2dc30a4961a4b50a40f5c706d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:05.567019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7908f2dc30a4961a4b50a40f5c706d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:05.568917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:05.581203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7908f2dc30a4961a4b50a40f5c706d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:05.581856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7908f2dc30a4961a4b50a40f5c706d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:05.589601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7908f2dc30a4961a4b50a40f5c706d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:05.595421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1151/2280] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:05 2022] DELETE /identity/v3/users/a7908f2dc30a4961a4b50a40f5c706d0 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:05.599961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:05.600461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:05.600570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:05.618750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:05.619971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:05.682083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1130/2281] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:05.692175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:05.693599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:05.750817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21a3a26-d751-45d6-995a-b620bf56cbc1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:05.754464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7753d3-3ca0-4797-b5c0-550c1e986d77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:05.755106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7753d3-3ca0-4797-b5c0-550c1e986d77 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:05.755391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7753d3-3ca0-4797-b5c0-550c1e986d77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:05.755663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7753d3-3ca0-4797-b5c0-550c1e986d77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:05.765215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7753d3-3ca0-4797-b5c0-550c1e986d77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:05.766640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7753d3-3ca0-4797-b5c0-550c1e986d77 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:05.776601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1152/2282] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:06.039092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.040553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:06.053233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ade8a-bfe3-4c24-a1a3-4d1a1266cac3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.058591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.059301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.059592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.059867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] PATH_INFO: `/v3/projects/b3b2f87c566844beb02d98fa01082bfe` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.061017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3b2f87c566844beb02d98fa01082bfe)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.063008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.113529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3b2f87c566844beb02d98fa01082bfe', 'request_id': 'req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:06.122058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3b2f87c566844beb02d98fa01082bfe', 'request_id': 'req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:06.128232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3b2f87c566844beb02d98fa01082bfe', 'request_id': 'req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:06.140634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] The token cache is being invalidate because project b3b2f87c566844beb02d98fa01082bfe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:06.140927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:06.141590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-1a7eaaa5-7cf0-41bf-a72c-c5d91f97590f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:06.143132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1131/2283] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:06 2022] DELETE /identity/v3/projects/b3b2f87c566844beb02d98fa01082bfe => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:06.148350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.148638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.148947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.170440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:06.171695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:06.232440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1153/2284] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:06.240258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.240545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.240822 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.261181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:06.262375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:06.270299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1132/2285] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:06.274489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.284287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7ea84-e833-4f6a-a142-685b96d13a89 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.288880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.289475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.289756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.290031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.291200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:44:06.291505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.293091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.304430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1154/2286] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:44:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:44:06.309226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.318638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3d616-5e36-4500-9269-2ebe68862221 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.323524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.324107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.324385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.324655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.325287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.327195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.355180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1133/2287] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:06.359093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.369578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2cd7bb-b228-4f27-9378-03399e8d378a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.424131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.424727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.425009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.425284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.425922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.427871 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.438288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:06.452237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1155/2288] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/users => generated 287 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:06.456300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.466509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a10b1c1-4d54-4b02-a633-35eedd670c7c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.471215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.471811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.472090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.472360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.473002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.474603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.484841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1134/2289] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:06.488600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.498718 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5a67a6-4517-4801-9de1-cff298e3639a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.503265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e114198b-42d5-44ed-a3be-95ec9d7ef14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.503844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114198b-42d5-44ed-a3be-95ec9d7ef14d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.504125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114198b-42d5-44ed-a3be-95ec9d7ef14d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.504397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114198b-42d5-44ed-a3be-95ec9d7ef14d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.505026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114198b-42d5-44ed-a3be-95ec9d7ef14d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.506578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114198b-42d5-44ed-a3be-95ec9d7ef14d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.516880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1156/2290] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:06.520726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.530944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1918537-7ba9-42d5-9f37-b34a222ed63c admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.535541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.536171 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.536452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.536728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] PATH_INFO: `/v3/projects/2cae0d54b37c4f8799d9c55b7af9454e/users/fb746615d23f41c58426706caed8e907/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.551379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cae0d54b37c4f8799d9c55b7af9454e, user_id=fb746615d23f41c58426706caed8e907, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.554075 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.566457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1135/2291] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:06 2022] PUT /identity/v3/projects/2cae0d54b37c4f8799d9c55b7af9454e/users/fb746615d23f41c58426706caed8e907/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:06.576625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.576910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.577182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.605856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] MFA Rules not processed for user `fb746615d23f41c58426706caed8e907`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:06.607106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:06.670761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1157/2292] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:06.675346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.684290 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:06.738243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d640f417-a339-41af-a784-418611da6c45 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.742898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.743534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.743820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.744122 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.744770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.746683 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.772874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1136/2293] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/projects => generated 371 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:06.777307 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.787677 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ae254-bcb5-4fbb-9b8e-1b3b4ef4a45c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.842387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.843094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.843380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.843652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.844280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.846173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.867931 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1158/2294] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:06 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:06.871919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.881878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce681cfa-e495-4186-98a8-611c87b9b8d6 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.887047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.887638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.887911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.888180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.888840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.890418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.901384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1137/2295] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:06.905421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.915238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af53727-eda8-4242-9d50-e8cb2ff24350 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:06.919861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:06.920506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:06.920780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:06.921050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] PATH_INFO: `/v3/projects/44119576be4a4df3bf473d2d557f37d3/users/9339c9e169644b47926cec25cceb4f18/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:06.935383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] RBAC: Authorizing `identity:create_grant(project_id=44119576be4a4df3bf473d2d557f37d3, user_id=9339c9e169644b47926cec25cceb4f18, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:06.938009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:06.951131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1159/2296] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:06 2022] PUT /identity/v3/projects/44119576be4a4df3bf473d2d557f37d3/users/9339c9e169644b47926cec25cceb4f18/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:06.955160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:06.965670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2ffa8e-d2f7-4db2-867a-a8a0bbcb3778 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:07.019632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc99069d-4bc3-4829-a9eb-7c059fa99566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:07.020238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc99069d-4bc3-4829-a9eb-7c059fa99566 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.020518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc99069d-4bc3-4829-a9eb-7c059fa99566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.020797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc99069d-4bc3-4829-a9eb-7c059fa99566 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.021437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc99069d-4bc3-4829-a9eb-7c059fa99566 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:07.023059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc99069d-4bc3-4829-a9eb-7c059fa99566 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:07.033259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1138/2297] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:06 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:07.037264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:07.047382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494f1f94-64b8-4833-ae45-565635ca14de admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:07.052060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:07.052700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.052980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.053252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] PATH_INFO: `/v3/projects/44119576be4a4df3bf473d2d557f37d3/users/9339c9e169644b47926cec25cceb4f18/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.055180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=44119576be4a4df3bf473d2d557f37d3, user_id=9339c9e169644b47926cec25cceb4f18, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:07.057814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:07.067998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1160/2298] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:07 2022] PUT /identity/v3/projects/44119576be4a4df3bf473d2d557f37d3/users/9339c9e169644b47926cec25cceb4f18/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:07.079612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.079904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.080182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.107830 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] MFA Rules not processed for user `9339c9e169644b47926cec25cceb4f18`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:07.109028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:07.176246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1139/2299] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:07 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:07.187581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:07.189270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:07.218310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:07.249192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7b6818-c962-4506-a473-913db4f77b7b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:07.253140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246baf03-a260-4090-bfa9-8cc6fc98ce73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:07.253742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246baf03-a260-4090-bfa9-8cc6fc98ce73 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.254021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246baf03-a260-4090-bfa9-8cc6fc98ce73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.254291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246baf03-a260-4090-bfa9-8cc6fc98ce73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.264021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246baf03-a260-4090-bfa9-8cc6fc98ce73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:07.265501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246baf03-a260-4090-bfa9-8cc6fc98ce73 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:07.274815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1161/2300] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:07 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:07.654428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:07.658672 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:07.717693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3041fa84-834e-4ccb-8f95-6c8f5d878e9e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:07.722183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:07.722813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.723124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.723399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] PATH_INFO: `/v3/users/fb746615d23f41c58426706caed8e907` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.724513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb746615d23f41c58426706caed8e907)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:07.726466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:07.773984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb746615d23f41c58426706caed8e907'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:07.780745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb746615d23f41c58426706caed8e907'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:07.793256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb746615d23f41c58426706caed8e907'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:07.793878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb746615d23f41c58426706caed8e907'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:07.800653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb746615d23f41c58426706caed8e907'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:07.806999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1140/2301] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:07 2022] DELETE /identity/v3/users/fb746615d23f41c58426706caed8e907 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:07.811593 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.812071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.812169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.831120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:07.832309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:07.889978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1162/2302] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:07.899590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:07.901035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:07.957800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dd5c6f-ed51-4ccb-8e19-d49227169100 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:07.961420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:07.962050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:07.962327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:07.962600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:07.971717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:07.973140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:07.982873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1141/2303] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:08.247946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:08.249366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:08.262457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc34fa-b04a-4a05-96f3-af596aa564c7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:08.272287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:08.272937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:08.273215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:08.273486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] PATH_INFO: `/v3/projects/2cae0d54b37c4f8799d9c55b7af9454e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:08.274626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2cae0d54b37c4f8799d9c55b7af9454e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:08.277142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:08.324668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2cae0d54b37c4f8799d9c55b7af9454e', 'request_id': 'req-ecf28156-8a2c-44cd-ac59-206513002b04'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.331417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2cae0d54b37c4f8799d9c55b7af9454e', 'request_id': 'req-ecf28156-8a2c-44cd-ac59-206513002b04'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.337763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2cae0d54b37c4f8799d9c55b7af9454e', 'request_id': 'req-ecf28156-8a2c-44cd-ac59-206513002b04'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.348795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] The token cache is being invalidate because project 2cae0d54b37c4f8799d9c55b7af9454e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:08.349101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.349710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.351244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1163/2304] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:08 2022] DELETE /identity/v3/projects/2cae0d54b37c4f8799d9c55b7af9454e => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:08.355959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:08.365500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64cbae-581b-44d9-87ef-1d64fb7fb960 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:08.385458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:08.423899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:08.424526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:08.424804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:08.425076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] PATH_INFO: `/v3/users/9339c9e169644b47926cec25cceb4f18` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:08.426188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=9339c9e169644b47926cec25cceb4f18)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:08.428149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:08.474142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9339c9e169644b47926cec25cceb4f18'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.481405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9339c9e169644b47926cec25cceb4f18'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.493670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9339c9e169644b47926cec25cceb4f18'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.494300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9339c9e169644b47926cec25cceb4f18'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.501104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-734d49ca-be75-4909-a657-3796dff718cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9339c9e169644b47926cec25cceb4f18'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.506850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1142/2305] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:08 2022] DELETE /identity/v3/users/9339c9e169644b47926cec25cceb4f18 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:08.774205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:08.776404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:08.789276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf28156-8a2c-44cd-ac59-206513002b04 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:08.836250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:08.847332 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:08.847963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:08.848241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:08.848513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] PATH_INFO: `/v3/projects/44119576be4a4df3bf473d2d557f37d3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:08.849625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44119576be4a4df3bf473d2d557f37d3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:08.851625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:08.896868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44119576be4a4df3bf473d2d557f37d3', 'request_id': 'req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.904147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44119576be4a4df3bf473d2d557f37d3', 'request_id': 'req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.909804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44119576be4a4df3bf473d2d557f37d3', 'request_id': 'req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.920615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] The token cache is being invalidate because project 44119576be4a4df3bf473d2d557f37d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:08.920905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.921529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfc4b15c-f8ea-479e-bb3b-94c9d06d3e73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:08.923040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1164/2306] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:08 2022] DELETE /identity/v3/projects/44119576be4a4df3bf473d2d557f37d3 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:08.927826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:08.928310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:08.928410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:08.950376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:08.951604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:09.009831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1143/2307] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:09.017782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.018070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.018346 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.040911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:09.042117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:09.050341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1165/2308] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:09.054591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.064638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80779e59-4d4e-495b-950c-ad5a1566497f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.069513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.070103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.070378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.070645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.071315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:44:09.072269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.073852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.086007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1144/2309] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:44:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:44:09.090389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.099486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a453a211-8841-4f6e-b81e-70e61ceea484 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.104103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.104697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.105001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.105276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.105911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.107843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.135213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1166/2310] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/projects => generated 357 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:09.139413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.149745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45204b7-9976-4a1e-938b-07780ed516e2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.178246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:09.207630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.208238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.208514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.208787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.209423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.211375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.233209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1145/2311] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/users => generated 279 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:09.237486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.248567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c50f23-7823-47d7-852f-06a58e98723d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.253254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.253851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.254126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.254394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.255090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.256644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.266785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1167/2312] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:09.270829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.281313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33557dcb-0b2c-47fd-8430-58a5a23bfa0b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.285897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1eff71-7e79-471e-876b-25cfe5130a36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.286492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1eff71-7e79-471e-876b-25cfe5130a36 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.286774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1eff71-7e79-471e-876b-25cfe5130a36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.287090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1eff71-7e79-471e-876b-25cfe5130a36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.287738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1eff71-7e79-471e-876b-25cfe5130a36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.289333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1eff71-7e79-471e-876b-25cfe5130a36 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.302833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1146/2313] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:09.307661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.318024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06a68d9-592b-4c30-90fa-2348d0cb2763 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.322616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.323297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.323579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.323853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] PATH_INFO: `/v3/projects/2c8cb326178f4d5881d50cf9f7cd3729/users/80d7fd5f53674f8da6b813f806ba4762/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.338352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c8cb326178f4d5881d50cf9f7cd3729, user_id=80d7fd5f53674f8da6b813f806ba4762, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.341079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.355103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1168/2314] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:09 2022] PUT /identity/v3/projects/2c8cb326178f4d5881d50cf9f7cd3729/users/80d7fd5f53674f8da6b813f806ba4762/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:44:09.365228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.365562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.365837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.394722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] MFA Rules not processed for user `80d7fd5f53674f8da6b813f806ba4762`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:09.395962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:09.460499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1147/2315] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:09.465025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.526001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb8b3f0-622d-4511-8e62-3f76480ed952 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.530655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.531287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.531573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.531882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.532511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.534422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.561191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1169/2316] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/projects => generated 357 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:09.565231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.576865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569af5f1-6389-437c-ac49-a8729b263836 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.632865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.633399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.633682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.633953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.635135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.637045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.658117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1148/2317] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:09.662148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.667865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:09.674401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce97f1ec-2eb4-463e-a609-930f5dbf41e5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.680378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.681917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.681917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.681917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.682694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.684333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.696006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1170/2318] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:09.700926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.710920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca8f2e4-9e83-487b-8643-548e6e367770 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.715513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.716156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.716434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.716709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] PATH_INFO: `/v3/projects/941915a1bdf64e68a16e3c4b3f959e54/users/8b3bcad38386496aacb13b0c412335df/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.731083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=941915a1bdf64e68a16e3c4b3f959e54, user_id=8b3bcad38386496aacb13b0c412335df, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.733742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.747005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1149/2319] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:09 2022] PUT /identity/v3/projects/941915a1bdf64e68a16e3c4b3f959e54/users/8b3bcad38386496aacb13b0c412335df/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:09.751015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.761859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bc7197-2cf8-4b61-b012-261f6044cabf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.816865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc82fc87-2d8c-4ff8-a94d-ad71fe851295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.817492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc82fc87-2d8c-4ff8-a94d-ad71fe851295 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.817770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc82fc87-2d8c-4ff8-a94d-ad71fe851295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.818043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc82fc87-2d8c-4ff8-a94d-ad71fe851295 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.818675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc82fc87-2d8c-4ff8-a94d-ad71fe851295 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.820282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc82fc87-2d8c-4ff8-a94d-ad71fe851295 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.830359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1171/2320] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:09 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:09.834493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.841049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:09.845340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccac07f-07c0-4730-8eb5-ec7b713aea98 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:09.851069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:09.851747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.852022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.852292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] PATH_INFO: `/v3/projects/941915a1bdf64e68a16e3c4b3f959e54/users/8b3bcad38386496aacb13b0c412335df/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.854204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=941915a1bdf64e68a16e3c4b3f959e54, user_id=8b3bcad38386496aacb13b0c412335df, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:09.856905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:09.867211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1150/2321] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:09 2022] PUT /identity/v3/projects/941915a1bdf64e68a16e3c4b3f959e54/users/8b3bcad38386496aacb13b0c412335df/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:09.878778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:09.879322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:09.879420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:09.906656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] MFA Rules not processed for user `8b3bcad38386496aacb13b0c412335df`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:09.907903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:09.973391 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1172/2322] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:09 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:09.985128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:09.986868 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:10.043499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0531813-9037-4d74-971c-969c9c652e9e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:10.047177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:10.047772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:10.048052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:10.048322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:10.057662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:10.059450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:10.068923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1151/2323] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:09 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:10.143474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:10.152995 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32076fb-c852-4962-b5c7-f75a08da6195 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:10.156539 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:10.157182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:10.157409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:10.157682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:10.213731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:10.215220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:10.223988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1173/2324] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:10 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:13.844179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:13.845873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:13.917440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971a0588-94ca-499b-a115-0e71cdf7af2c service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:13.923139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:13.923739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:13.924016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:13.924285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:13.925999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:13.985475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:13.987108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:13.997428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1152/2325] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:13 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:20.219989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:21.579946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:21.582786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:21.614956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26f3e73-de7c-413e-8b39-20767e4dfe8c service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:21.614956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:21.614956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:21.614956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:21.615959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:21.615959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:21.679604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:21.681397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8740e8a8-82f6-4385-9b77-bfbcf6ff723b service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:21.691779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1174/2326] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:21 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:23.993100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=85,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:25.076388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:25.141215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5809475-48c0-4ccb-84ff-fd6a53645929 service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:25.145760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:25.146383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:25.146663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:25.146951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] PATH_INFO: `/v3/users/80d7fd5f53674f8da6b813f806ba4762` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:25.148092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80d7fd5f53674f8da6b813f806ba4762)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:25.150006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:25.200720 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80d7fd5f53674f8da6b813f806ba4762'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.207814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80d7fd5f53674f8da6b813f806ba4762'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.220095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80d7fd5f53674f8da6b813f806ba4762'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.220726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80d7fd5f53674f8da6b813f806ba4762'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.227455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80d7fd5f53674f8da6b813f806ba4762'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.233287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1153/2327] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:25 2022] DELETE /identity/v3/users/80d7fd5f53674f8da6b813f806ba4762 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:44:25.237859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:25.238368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:25.238471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:25.257602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:25.258796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:25.318133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1175/2328] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:25.328353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:25.329740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:25.385779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed813073-4882-4008-b666-02dcc7de4544 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:25.390146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:25.390750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:25.391062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:25.391337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:25.400252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:25.401677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:25.411185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1154/2329] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:25.676059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:25.678064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:25.694500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ebd446-ebdd-4058-8ca1-1deff48c4323 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:25.704914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:25.705577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:25.705861 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:25.706138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] PATH_INFO: `/v3/projects/2c8cb326178f4d5881d50cf9f7cd3729` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:25.708182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c8cb326178f4d5881d50cf9f7cd3729)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:25.710099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:25.765780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c8cb326178f4d5881d50cf9f7cd3729', 'request_id': 'req-e960a8d1-d4a7-44c9-98b4-671eddf7b479'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.773132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c8cb326178f4d5881d50cf9f7cd3729', 'request_id': 'req-e960a8d1-d4a7-44c9-98b4-671eddf7b479'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.778823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c8cb326178f4d5881d50cf9f7cd3729', 'request_id': 'req-e960a8d1-d4a7-44c9-98b4-671eddf7b479'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.789736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] The token cache is being invalidate because project 2c8cb326178f4d5881d50cf9f7cd3729 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:25.790027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.790652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.792151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1176/2330] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:25 2022] DELETE /identity/v3/projects/2c8cb326178f4d5881d50cf9f7cd3729 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:25.796787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:25.806197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f5243f-9aa2-4da5-ad22-69564e86235e service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:25.824714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:25.864271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:25.864906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:25.865168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:25.865437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] PATH_INFO: `/v3/users/8b3bcad38386496aacb13b0c412335df` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:25.866646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b3bcad38386496aacb13b0c412335df)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:25.868595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:25.913074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b3bcad38386496aacb13b0c412335df'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.921086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b3bcad38386496aacb13b0c412335df'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.933042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b3bcad38386496aacb13b0c412335df'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.934272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b3bcad38386496aacb13b0c412335df'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.941403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-df195806-a768-4f6c-9ed8-3024bf72bfa9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b3bcad38386496aacb13b0c412335df'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:25.947189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1155/2331] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:25 2022] DELETE /identity/v3/users/8b3bcad38386496aacb13b0c412335df => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:26.226570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.228127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:26.244083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e960a8d1-d4a7-44c9-98b4-671eddf7b479 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.268323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:26.312169 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.313069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.313348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.313622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] PATH_INFO: `/v3/projects/941915a1bdf64e68a16e3c4b3f959e54` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.314751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=941915a1bdf64e68a16e3c4b3f959e54)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.316945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.367157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '941915a1bdf64e68a16e3c4b3f959e54', 'request_id': 'req-028b8805-9af2-4e11-9896-4ba8924a9e0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:26.375074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '941915a1bdf64e68a16e3c4b3f959e54', 'request_id': 'req-028b8805-9af2-4e11-9896-4ba8924a9e0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:26.380927 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '941915a1bdf64e68a16e3c4b3f959e54', 'request_id': 'req-028b8805-9af2-4e11-9896-4ba8924a9e0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:26.392424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] The token cache is being invalidate because project 941915a1bdf64e68a16e3c4b3f959e54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:26.392719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:26.393361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-028b8805-9af2-4e11-9896-4ba8924a9e0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:26.394886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1177/2332] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:26 2022] DELETE /identity/v3/projects/941915a1bdf64e68a16e3c4b3f959e54 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:26.400312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.400820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.400926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.420694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:26.421916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:26.480992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1156/2333] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:26.489590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.489877 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.490157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.510630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:26.512364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:26.520478 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1178/2334] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:44:26.524828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.535200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84997227-3b6f-409e-a970-8e7c3cfefc65 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.539717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.540312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.540591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.540863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.541502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:44:26.541803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.543545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.554952 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1157/2335] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:44:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:44:26.559249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.568644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb1607-8d3e-4cf2-9be7-23791a4dcfc1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.573287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.574373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.574651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.574922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.575591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.577491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.604291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1179/2336] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:26 2022] POST /identity/v3/projects => generated 365 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:26.608281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.618015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffaa67d-8d6e-403d-aaad-389120090247 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.643595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:26.676641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.677245 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.677525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.677801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.678442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.680392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.701607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1158/2337] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:26 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:26.705679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.717095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe964a6a-3c59-4327-a91f-351c42eab5a1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.722897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.723532 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.723812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.724101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.724711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.726285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.736405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1180/2338] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:44:26.740413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.752973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56441102-3bca-40af-a594-5a8320f2c215 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.756736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b9f5c6-27b5-4a4a-86b7-4a8dfb2921b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.757340 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b9f5c6-27b5-4a4a-86b7-4a8dfb2921b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.757616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b9f5c6-27b5-4a4a-86b7-4a8dfb2921b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.757885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b9f5c6-27b5-4a4a-86b7-4a8dfb2921b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.758557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b9f5c6-27b5-4a4a-86b7-4a8dfb2921b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.760175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b9f5c6-27b5-4a4a-86b7-4a8dfb2921b0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.771704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1159/2339] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:26.776224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.786345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8f945-ce20-4fbe-b6b8-9c41003ffa5d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:26.790979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:26.791629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.791915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.792194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] PATH_INFO: `/v3/projects/544763d1eed549bdbc245e665822df9f/users/a09316b9083f4608b8811d772c058555/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.806784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=544763d1eed549bdbc245e665822df9f, user_id=a09316b9083f4608b8811d772c058555, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:26.809974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:26.824522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1181/2340] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:26 2022] PUT /identity/v3/projects/544763d1eed549bdbc245e665822df9f/users/a09316b9083f4608b8811d772c058555/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:26.835782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:26.836074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:26.836356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:26.863368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] MFA Rules not processed for user `a09316b9083f4608b8811d772c058555`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:26.864504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:26.927079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1160/2341] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:26.931672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:26.981953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:26.997743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02b4b1-e219-4e78-b6f5-81a28913c0a0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.002425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.003059 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.003344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.003620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.004261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.006159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.032494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1182/2342] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:26 2022] POST /identity/v3/projects => generated 363 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:27.036454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.046856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b6936-4f4f-4835-855e-065d0837b49f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.102001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.102599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.102880 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.103184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.103821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.106277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.127243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1161/2343] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:27 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:27.131168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.141601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc022ec-a9ac-4235-98ff-c8e63192cceb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.146185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.146786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.147123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.147389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.148062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.149687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.160245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1183/2344] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:27.164132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.174915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8a7e9-5af9-43ef-ab2d-6f2684e8b620 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.179545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.180189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.180476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.180746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] PATH_INFO: `/v3/projects/37a86e3e96fb4a9cb5fa11435ea61f23/users/dcd6d297a3f64d56b5edce53589a0727/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.197047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37a86e3e96fb4a9cb5fa11435ea61f23, user_id=dcd6d297a3f64d56b5edce53589a0727, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.199872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.212035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1162/2345] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:27 2022] PUT /identity/v3/projects/37a86e3e96fb4a9cb5fa11435ea61f23/users/dcd6d297a3f64d56b5edce53589a0727/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:27.215937 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.225730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d89ac92-19b0-4a4b-aec3-de46a98a46a0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.288037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0cb975-e967-4a49-8f1b-6269e8daa38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.288717 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0cb975-e967-4a49-8f1b-6269e8daa38f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.288911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0cb975-e967-4a49-8f1b-6269e8daa38f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.289182 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0cb975-e967-4a49-8f1b-6269e8daa38f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.289821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0cb975-e967-4a49-8f1b-6269e8daa38f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.291452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0cb975-e967-4a49-8f1b-6269e8daa38f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.301695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1184/2346] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:27 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:44:27.306033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.312182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:27.318718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df8631-02f4-4b3e-9d19-89d0ecd7a214 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.323521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.324190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.324465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.324733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] PATH_INFO: `/v3/projects/37a86e3e96fb4a9cb5fa11435ea61f23/users/dcd6d297a3f64d56b5edce53589a0727/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.326596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37a86e3e96fb4a9cb5fa11435ea61f23, user_id=dcd6d297a3f64d56b5edce53589a0727, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.329225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.339814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1163/2347] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:27 2022] PUT /identity/v3/projects/37a86e3e96fb4a9cb5fa11435ea61f23/users/dcd6d297a3f64d56b5edce53589a0727/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:27.350082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.350710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.350846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.382071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] MFA Rules not processed for user `dcd6d297a3f64d56b5edce53589a0727`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:27.383260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:27.454363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1185/2348] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:27 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:27.465790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.467608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:27.525690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a3858c-8eec-491a-8015-0bb72f9fc476 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.530172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cb23d1-26bc-4ac6-a842-295feee21d6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.530858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cb23d1-26bc-4ac6-a842-295feee21d6c service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.531220 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cb23d1-26bc-4ac6-a842-295feee21d6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.531497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cb23d1-26bc-4ac6-a842-295feee21d6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.540869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cb23d1-26bc-4ac6-a842-295feee21d6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.542305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cb23d1-26bc-4ac6-a842-295feee21d6c service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.552399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1164/2349] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:27 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:27.625555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:27.635272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e9aa96-2452-4e7a-b11e-cf09295a0809 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:27.639281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:27.639876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:27.640161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:27.640437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:27.684425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:27.701963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:27.703482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:27.712846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1186/2350] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:27 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:30.174056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:30.174566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:30.174681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:30.194251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:30.195423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:30.250594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1165/2351] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:44:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:30.297348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:30.298767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:30.375979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dbd4a9-90ab-403c-a718-c6ad1454ac91 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:30.381632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceed711-b40d-4bf3-95c8-358f9db73bc5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:30.382235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceed711-b40d-4bf3-95c8-358f9db73bc5 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:30.382517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceed711-b40d-4bf3-95c8-358f9db73bc5 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:30.382790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceed711-b40d-4bf3-95c8-358f9db73bc5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:30.391609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceed711-b40d-4bf3-95c8-358f9db73bc5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:30.393256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceed711-b40d-4bf3-95c8-358f9db73bc5 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:30.402674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1187/2352] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:44:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:44:30.430714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:30.555490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47f085c-df7d-4654-a340-8fbd988e302d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:30.561750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:30.562503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:30.563090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:30.563465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] PATH_INFO: `/v3/users/a09316b9083f4608b8811d772c058555` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:30.565451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a09316b9083f4608b8811d772c058555)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:30.568219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:30.626653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:30.636223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a09316b9083f4608b8811d772c058555'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:30.664077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a09316b9083f4608b8811d772c058555'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:30.686478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a09316b9083f4608b8811d772c058555'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:30.690980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a09316b9083f4608b8811d772c058555'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:30.707247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a09316b9083f4608b8811d772c058555'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:30.714173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1166/2353] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:30 2022] DELETE /identity/v3/users/a09316b9083f4608b8811d772c058555 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:30.719256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:30.719784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:30.720269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:30.743255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:30.744900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:30.815449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1188/2354] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:30.829018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:30.830677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:30.904815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e3484-5651-49e4-bb29-3a408e888074 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:30.908731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:30.909459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:30.909803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:30.910145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:30.920390 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:30.923226 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:30.933289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1167/2355] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:31.416752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:31.418426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:31.443238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcff0df5-4f38-40c9-8bc2-db34b0bd4b01 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:31.461252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:31.461252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:31.461252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:31.461628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] PATH_INFO: `/v3/projects/544763d1eed549bdbc245e665822df9f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:31.462921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] RBAC: Authorizing `identity:delete_project(project_id=544763d1eed549bdbc245e665822df9f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:31.465007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:31.531326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '544763d1eed549bdbc245e665822df9f', 'request_id': 'req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.540377 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '544763d1eed549bdbc245e665822df9f', 'request_id': 'req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.546892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '544763d1eed549bdbc245e665822df9f', 'request_id': 'req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.549516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:31.564930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] The token cache is being invalidate because project 544763d1eed549bdbc245e665822df9f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:31.564930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.566064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.571556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1189/2356] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:31 2022] DELETE /identity/v3/projects/544763d1eed549bdbc245e665822df9f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:31.573050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:31.584141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06812244-7003-404b-9dfb-43045f788826 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:31.686252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:31.686252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:31.686252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:31.686252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] PATH_INFO: `/v3/users/dcd6d297a3f64d56b5edce53589a0727` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:31.688318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dcd6d297a3f64d56b5edce53589a0727)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:31.690410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:31.771479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dcd6d297a3f64d56b5edce53589a0727'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.782924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dcd6d297a3f64d56b5edce53589a0727'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.790278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:31.801737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dcd6d297a3f64d56b5edce53589a0727'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.802884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dcd6d297a3f64d56b5edce53589a0727'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.818807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-875a84fe-f52c-4f6f-b178-e874ad567317 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dcd6d297a3f64d56b5edce53589a0727'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:31.826481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1168/2357] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:31 2022] DELETE /identity/v3/users/dcd6d297a3f64d56b5edce53589a0727 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:32.303353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:32.305725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:32.326242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab4f9c-f1f1-4e34-8612-a9a47ead3874 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:32.448261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:32.448963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:32.449218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:32.450984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] PATH_INFO: `/v3/projects/37a86e3e96fb4a9cb5fa11435ea61f23` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:32.453201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37a86e3e96fb4a9cb5fa11435ea61f23)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:32.455236 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:32.531323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37a86e3e96fb4a9cb5fa11435ea61f23', 'request_id': 'req-ac2f8fab-c60f-4eca-ba36-95543a2c8924'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:32.544664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37a86e3e96fb4a9cb5fa11435ea61f23', 'request_id': 'req-ac2f8fab-c60f-4eca-ba36-95543a2c8924'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:32.554376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37a86e3e96fb4a9cb5fa11435ea61f23', 'request_id': 'req-ac2f8fab-c60f-4eca-ba36-95543a2c8924'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:32.569688 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] The token cache is being invalidate because project 37a86e3e96fb4a9cb5fa11435ea61f23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:32.570287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:32.571186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac2f8fab-c60f-4eca-ba36-95543a2c8924 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:32.573110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1190/2358] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:32 2022] DELETE /identity/v3/projects/37a86e3e96fb4a9cb5fa11435ea61f23 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:32.580366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:32.580914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:32.581187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:32.605670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:32.608043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:32.678432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1169/2359] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:32.696698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:32.696990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:32.697261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:32.723263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:32.724407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:32.737666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1191/2360] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:32.738916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:32.748971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b20956d-7350-4fc5-bdbe-c40f409c0fa9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:32.753815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:32.754578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:32.754954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:32.755576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:32.756313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:44:32.756694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:32.758380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:32.772253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1170/2361] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:44:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:44:32.776288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:32.787702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74db0ed6-0a2f-498e-becc-511b7c841cf0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:32.794215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:32.795393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:32.795794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:32.796402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:32.797128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:32.801043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:32.835143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1192/2362] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:32 2022] POST /identity/v3/projects => generated 369 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:32.839113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:32.864730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933f692c-71f6-4b2e-b3a6-fc9eaa82cb52 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:32.962853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:32.963553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:32.963992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:32.964272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:32.964926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:32.967079 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:32.999257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1171/2363] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:32 2022] POST /identity/v3/users => generated 285 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:33.000961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.014197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8630b3-958d-4111-bf63-e67924b858a4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.019505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.020268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.020555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.020830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.021486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.023113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.044966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1193/2364] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:32 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:33.049555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.066321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd0796-7695-4c15-949a-9f22247dc431 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.072482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc75bfe-940a-4dab-a26b-54bab1ead76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.073099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc75bfe-940a-4dab-a26b-54bab1ead76c admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.073377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc75bfe-940a-4dab-a26b-54bab1ead76c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.076088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc75bfe-940a-4dab-a26b-54bab1ead76c admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.076088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc75bfe-940a-4dab-a26b-54bab1ead76c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.076713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc75bfe-940a-4dab-a26b-54bab1ead76c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.093421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1172/2365] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:33 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:33.095913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.114258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1c9f9-881d-4a7c-8463-de2191e566d0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.127196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.127867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.128147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.128420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] PATH_INFO: `/v3/projects/4f119a86a1844dbf9527dedad233de36/users/e0bbbdce8bba4a5fbc83bbb2d901e647/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.145542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f119a86a1844dbf9527dedad233de36, user_id=e0bbbdce8bba4a5fbc83bbb2d901e647, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.148243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.163790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1194/2366] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:33 2022] PUT /identity/v3/projects/4f119a86a1844dbf9527dedad233de36/users/e0bbbdce8bba4a5fbc83bbb2d901e647/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:33.175233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.175665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.175932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.214169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] MFA Rules not processed for user `e0bbbdce8bba4a5fbc83bbb2d901e647`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:33.215433 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:33.276440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:33.325964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1173/2367] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:33.330486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.369450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:33.417122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9811aeb-bfdd-4748-8816-99470c0f869f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.422372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.423151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.423432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.423704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.424339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.426364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.463837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1195/2368] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:33 2022] POST /identity/v3/projects => generated 367 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:33.467977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.481392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d04b477-b9f8-4ff3-a0d1-20d42c8960ef None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.554465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.556797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.557089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.557368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.558030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.561024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.603453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1174/2369] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:33 2022] POST /identity/v3/users => generated 290 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:33.604290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.634416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7408ec5d-af90-4895-91a9-a540a81d0d7b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.642550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.643287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.643649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.644164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.644900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.646733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.661174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1196/2370] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:33 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:44:33.665431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.676093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5320017-f9da-4fc6-b1f9-cb329e43f4cc admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.683511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.684236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.685024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.685297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] PATH_INFO: `/v3/projects/e5efd5d5b6104f7da3355398564d14dd/users/e7d4b643a8a5489c907e3a3ce52e9d35/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.704374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5efd5d5b6104f7da3355398564d14dd, user_id=e7d4b643a8a5489c907e3a3ce52e9d35, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.707384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.741871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1175/2371] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:33 2022] PUT /identity/v3/projects/e5efd5d5b6104f7da3355398564d14dd/users/e7d4b643a8a5489c907e3a3ce52e9d35/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:33.747723 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.761852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84cdc89-aade-4f18-a794-1695a4f0adf3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.834618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f08bb-9463-4253-893a-039e524a43ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.835529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f08bb-9463-4253-893a-039e524a43ef admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.836084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f08bb-9463-4253-893a-039e524a43ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.836520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f08bb-9463-4253-893a-039e524a43ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.837309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f08bb-9463-4253-893a-039e524a43ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.839329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f08bb-9463-4253-893a-039e524a43ef admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.863167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1197/2372] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:33 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:44:33.871582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:33.902357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359021be-5e99-44d7-bb27-32c4e105b927 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:33.908386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:33.909175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.909483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.909771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] PATH_INFO: `/v3/projects/e5efd5d5b6104f7da3355398564d14dd/users/e7d4b643a8a5489c907e3a3ce52e9d35/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.912005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5efd5d5b6104f7da3355398564d14dd, user_id=e7d4b643a8a5489c907e3a3ce52e9d35, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:33.914639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:33.938308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1176/2373] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:33 2022] PUT /identity/v3/projects/e5efd5d5b6104f7da3355398564d14dd/users/e7d4b643a8a5489c907e3a3ce52e9d35/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:33.942035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:33.942248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:33.942546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:33.975010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] MFA Rules not processed for user `e7d4b643a8a5489c907e3a3ce52e9d35`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:33.976499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:34.105658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1198/2374] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:33 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:34.118610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:34.120363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:34.219578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff44eb1-3a5e-43ba-860b-0e5a49777be0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:34.229919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:34.230553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:34.230831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:34.231287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:34.244324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:34.248047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:34.258838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1177/2375] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:34 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:34.322435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:34.325984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:34.340817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cbb9d7-b135-4e4a-86c5-3b91e371eef9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:34.347590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c71a97-6d43-416d-bfc1-733eb0bc3d54 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:34.348445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c71a97-6d43-416d-bfc1-733eb0bc3d54 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:34.348728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c71a97-6d43-416d-bfc1-733eb0bc3d54 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:34.349001 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c71a97-6d43-416d-bfc1-733eb0bc3d54 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:34.365767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:34.420337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c71a97-6d43-416d-bfc1-733eb0bc3d54 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:34.421826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c71a97-6d43-416d-bfc1-733eb0bc3d54 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:34.431339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1199/2376] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:34 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:38.048988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:38.085827 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:38.129281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefe2fc3-e40c-4273-8684-7fb3261f9296 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:38.134163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:38.135016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:38.135299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:38.135573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] PATH_INFO: `/v3/users/e0bbbdce8bba4a5fbc83bbb2d901e647` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:38.138858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0bbbdce8bba4a5fbc83bbb2d901e647)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:38.138858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:38.200178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0bbbdce8bba4a5fbc83bbb2d901e647'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.211130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0bbbdce8bba4a5fbc83bbb2d901e647'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.225728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0bbbdce8bba4a5fbc83bbb2d901e647'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.226414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0bbbdce8bba4a5fbc83bbb2d901e647'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.240523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0bbbdce8bba4a5fbc83bbb2d901e647'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.246925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1178/2377] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:38 2022] DELETE /identity/v3/users/e0bbbdce8bba4a5fbc83bbb2d901e647 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:38.251604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:38.252141 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:38.253183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:38.281552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:38.282812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:38.384456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1200/2378] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:44:38.415626 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:38.422863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:38.509795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6425238-2185-438a-9b39-114fa63435ec admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:38.514676 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:38.515432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:38.515811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:38.516181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:38.528080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:38.528333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:38.554691 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1179/2379] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:38.889176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:38.890740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:38.904611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e332a8-7214-4b5c-a3c5-37969615ed2e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:38.921128 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:38.922052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:38.922410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:38.922885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] PATH_INFO: `/v3/projects/4f119a86a1844dbf9527dedad233de36` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:38.924427 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f119a86a1844dbf9527dedad233de36)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:38.926685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:38.978455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f119a86a1844dbf9527dedad233de36', 'request_id': 'req-423dd375-cfea-4a23-8bd0-84a8f3cb4063'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.989164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f119a86a1844dbf9527dedad233de36', 'request_id': 'req-423dd375-cfea-4a23-8bd0-84a8f3cb4063'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:38.996051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f119a86a1844dbf9527dedad233de36', 'request_id': 'req-423dd375-cfea-4a23-8bd0-84a8f3cb4063'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.008570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] The token cache is being invalidate because project 4f119a86a1844dbf9527dedad233de36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:39.009013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.009680 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.011471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1201/2380] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:38 2022] DELETE /identity/v3/projects/4f119a86a1844dbf9527dedad233de36 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:39.015277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:39.028533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd44466f-f731-43ee-8e98-dcb863a948c8 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:39.086172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:39.101551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:39.102288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:39.102832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:39.103232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] PATH_INFO: `/v3/users/e7d4b643a8a5489c907e3a3ce52e9d35` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:39.104479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7d4b643a8a5489c907e3a3ce52e9d35)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:39.107759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:39.172361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7d4b643a8a5489c907e3a3ce52e9d35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.186330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7d4b643a8a5489c907e3a3ce52e9d35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.203169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7d4b643a8a5489c907e3a3ce52e9d35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.203935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7d4b643a8a5489c907e3a3ce52e9d35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.212147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-077d4367-d849-4960-be4a-b45b3a9fd372 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7d4b643a8a5489c907e3a3ce52e9d35'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.222914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1180/2381] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:44:39 2022] DELETE /identity/v3/users/e7d4b643a8a5489c907e3a3ce52e9d35 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:39.578577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:39.586135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:39.613123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423dd375-cfea-4a23-8bd0-84a8f3cb4063 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:39.704825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:39.731781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:39.732578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:39.732951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:39.733434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] PATH_INFO: `/v3/projects/e5efd5d5b6104f7da3355398564d14dd` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:39.734678 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5efd5d5b6104f7da3355398564d14dd)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:39.736740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:39.820484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5efd5d5b6104f7da3355398564d14dd', 'request_id': 'req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.833711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5efd5d5b6104f7da3355398564d14dd', 'request_id': 'req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.842466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5efd5d5b6104f7da3355398564d14dd', 'request_id': 'req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.856862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] The token cache is being invalidate because project e5efd5d5b6104f7da3355398564d14dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:44:39.857250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.857989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-586d0fa5-c43f-41d0-902a-23bbed6fd1b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:44:39.859830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1202/2382] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:44:39 2022] DELETE /identity/v3/projects/e5efd5d5b6104f7da3355398564d14dd => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:39.864424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:39.864786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:39.865142 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:39.894723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:39.896856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:39.976321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1181/2383] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:39.986970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:39.986970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:39.986970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.008641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:40.008641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:40.020586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1203/2384] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:40.021378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.036160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa0ccd-3011-4c8a-ab2c-9cbfc7be4deb None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.043136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.044075 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.044605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.045051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.046466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:44:40.046973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.051259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.068094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1182/2385] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:44:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:44:40.073109 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.083797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76795c28-f9c3-4a90-9c22-4e65136d8ce1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.089244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.089950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.090506 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.091388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.092097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.094149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.136305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1204/2386] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:44:40 2022] POST /identity/v3/projects => generated 371 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:40.140861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.154715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56f554c-ca60-4c42-b05a-5548896e19a0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.232826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:40.243825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.245260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.246138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.247446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.248406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.251791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.299011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1183/2387] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:44:40 2022] POST /identity/v3/users => generated 286 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:44:40.302220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.327609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba82a21f-a42b-46a3-a5e5-b7a052391d5f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.345168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.346887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.347353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.347801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.348653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.350401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.361229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1205/2388] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:40 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:40.365215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.376577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e563a-4159-4802-93f1-d186a807e54b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.381352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fc8f9f-8527-4c0a-b4b1-4f5fe95107d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.381959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc8f9f-8527-4c0a-b4b1-4f5fe95107d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.382244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc8f9f-8527-4c0a-b4b1-4f5fe95107d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.382516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc8f9f-8527-4c0a-b4b1-4f5fe95107d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.383269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fc8f9f-8527-4c0a-b4b1-4f5fe95107d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.384877 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fc8f9f-8527-4c0a-b4b1-4f5fe95107d8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.396635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1184/2389] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:44:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:44:40.400814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.411694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1df078-f2e8-43ee-a660-fa967094d8fe admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.416769 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.417564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.417980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.418326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] PATH_INFO: `/v3/projects/3b1c0bde41ce41858402312e47ab5d73/users/1b746e9d1eb44e12ac5b279825e1a01d/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.437805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b1c0bde41ce41858402312e47ab5d73, user_id=1b746e9d1eb44e12ac5b279825e1a01d, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.443974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.459616 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1206/2390] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:44:40 2022] PUT /identity/v3/projects/3b1c0bde41ce41858402312e47ab5d73/users/1b746e9d1eb44e12ac5b279825e1a01d/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:44:40.528032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.528297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.528851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.563929 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] MFA Rules not processed for user `1b746e9d1eb44e12ac5b279825e1a01d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:40.565295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:40.633381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1185/2391] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:44:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:40.644912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:40.646638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:40.703473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb89878a-60ed-4fde-a631-c444d7527a52 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:40.707335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6937963-59f1-4baf-9c8c-88c1a54454b2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:40.708110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6937963-59f1-4baf-9c8c-88c1a54454b2 service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:40.708396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6937963-59f1-4baf-9c8c-88c1a54454b2 service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:40.708668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6937963-59f1-4baf-9c8c-88c1a54454b2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:40.717797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6937963-59f1-4baf-9c8c-88c1a54454b2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:40.719259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6937963-59f1-4baf-9c8c-88c1a54454b2 service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:40.728842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1207/2392] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:41.261519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:41.263625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:41.323978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc6e354-766c-4a1c-befc-11879587ba4e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:41.328784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:41.329472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:41.329935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:41.330260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:41.332064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:41.342641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:41.344120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744416e6-0526-4075-9cce-70b46e8c77db service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:41.353266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1186/2393] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:45.624697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e30041-8b42-45dd-b707-59f3d9e12009 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:45.625956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e30041-8b42-45dd-b707-59f3d9e12009 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:45.626517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e30041-8b42-45dd-b707-59f3d9e12009 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:45.628912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1208/2394] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:44:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:44:45.651751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:45.652628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:45.653690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:45.683645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:45.685360 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:45.763480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:45.776293 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e30041-8b42-45dd-b707-59f3d9e12009 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:45.778712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a2e30041-8b42-45dd-b707-59f3d9e12009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:45.795534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:45.795845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1187/2395] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:44:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:44:45.809896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:45.829339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c7edb-cb2a-48be-b60c-340fc5a26792 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:45.833128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d60b07-3985-4a4d-8bad-a1a967e3887e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:45.833876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d60b07-3985-4a4d-8bad-a1a967e3887e service glance] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:45.834596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d60b07-3985-4a4d-8bad-a1a967e3887e service glance] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:45.834954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d60b07-3985-4a4d-8bad-a1a967e3887e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:45.854889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d60b07-3985-4a4d-8bad-a1a967e3887e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:45.857789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d60b07-3985-4a4d-8bad-a1a967e3887e service glance] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:45.872170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1188/2396] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:45.889749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e30041-8b42-45dd-b707-59f3d9e12009 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:45.899977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:45.900639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:45.900949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:45.901220 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:45.903656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:45.916519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:45.918110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7f4936-acd2-4fc4-bec1-049c957b2456 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:45.929959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1209/2397] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:45.948194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69da3a3d-fd41-4dfc-bd5a-1a180c40e0ed None None] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:45.948487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69da3a3d-fd41-4dfc-bd5a-1a180c40e0ed None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:45.948771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69da3a3d-fd41-4dfc-bd5a-1a180c40e0ed None None] PATH_INFO: `/` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:45.950847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1189/2398] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:44:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:44:45.955452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ba996c-b32e-4fcc-84dc-4b59e6ef86b7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:45.955765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ba996c-b32e-4fcc-84dc-4b59e6ef86b7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:45.956077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ba996c-b32e-4fcc-84dc-4b59e6ef86b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:45.976859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f6ba996c-b32e-4fcc-84dc-4b59e6ef86b7 None None] MFA Rules not processed for user `4c501a6bee5b4f3a8973f727f9ef10cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:45.978100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f6ba996c-b32e-4fcc-84dc-4b59e6ef86b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:45.984753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1210/2399] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:44:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:45.996864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69da3a3d-fd41-4dfc-bd5a-1a180c40e0ed None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:46.009933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69da3a3d-fd41-4dfc-bd5a-1a180c40e0ed None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:46.015528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ead3ad-9db4-4b15-ab59-374c5f8753fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:46.016178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ead3ad-9db4-4b15-ab59-374c5f8753fc service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:46.016475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ead3ad-9db4-4b15-ab59-374c5f8753fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:46.016736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ead3ad-9db4-4b15-ab59-374c5f8753fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:46.024256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ead3ad-9db4-4b15-ab59-374c5f8753fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:46.025712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ead3ad-9db4-4b15-ab59-374c5f8753fc service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:46.034031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1190/2400] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:46.116846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee4b74-a17e-4413-b611-6e68c3038f3a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:46.117133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee4b74-a17e-4413-b611-6e68c3038f3a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:46.117683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee4b74-a17e-4413-b611-6e68c3038f3a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:46.119185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1211/2401] 10.222.0.20 () {54 vars in 921 bytes} [Fri Sep 9 04:44:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 09 04:44:46.125714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:46.126132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:46.126505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:46.146805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] MFA Rules not processed for user `0dd190c922964d51af263442650b427d`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:46.148076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:46.158019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1191/2402] 10.222.0.20 () {58 vars in 1018 bytes} [Fri Sep 9 04:44:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:46.168574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee4b74-a17e-4413-b611-6e68c3038f3a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:46.170287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-65ee4b74-a17e-4413-b611-6e68c3038f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:46.183361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee4b74-a17e-4413-b611-6e68c3038f3a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:46.187449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e21a2-20cd-4215-8961-f6f8fb2b73d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:46.187969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e21a2-20cd-4215-8961-f6f8fb2b73d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:46.188246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e21a2-20cd-4215-8961-f6f8fb2b73d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:46.188516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e21a2-20cd-4215-8961-f6f8fb2b73d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:46.198571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41e21a2-20cd-4215-8961-f6f8fb2b73d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:46.200662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41e21a2-20cd-4215-8961-f6f8fb2b73d0 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:46.211047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1212/2403] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:46.272206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:46.273612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:46.345856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49cbc62-49dd-488c-945a-b5b0129c978c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:46.350862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:46.351501 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:46.351779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:46.352050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:46.353789 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:46.364898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:46.366510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:46.376898 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1192/2404] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:44:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:44:49.723286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:49.723799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:49.723896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:49.749981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:44:49.751543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:49.810239 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1213/2405] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:44:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:44:49.823063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:44:49.824562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:44:49.894737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e436e-ce71-4623-9807-e468d63baa73 service placement] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:44:49.899874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:44:49.900487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:44:49.900773 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:44:49.901049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:44:49.910343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:44:49.911978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:44:49.921109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1193/2406] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:44:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:44:59.802647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=64 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:44:59.916919 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=79 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:06.701590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:06.729801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5c8c9d-a930-4eec-b6c0-ee37f33caa82 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:06.743529 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:06.744468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:06.744895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:06.745323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:06.748675 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:06.767611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:06.769935 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89c48b0-d21a-42d0-9555-4fae94524b8d service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:06.797359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1214/2407] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:07.447842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:07.460034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33857fa3-96ab-4924-aa61-62b28f748d09 service swift] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:07.466285 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:07.466888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:07.467198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:07.467471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:07.469248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:07.483346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:07.484861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:07.496525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1194/2408] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:08.620147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:08.620784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:08.621191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:08.645099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:08.646557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:08.656216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1215/2409] 10.222.0.20 () {58 vars in 967 bytes} [Fri Sep 9 04:45:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:08.669553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:08.681811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9aa5cb-90b7-49c8-833d-60aa7fdc853a service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:08.687344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1f3e0-61be-4aef-b26a-1aca0a12e60d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:08.687995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1f3e0-61be-4aef-b26a-1aca0a12e60d service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:08.688280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1f3e0-61be-4aef-b26a-1aca0a12e60d service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:08.688553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1f3e0-61be-4aef-b26a-1aca0a12e60d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:08.696947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb1f3e0-61be-4aef-b26a-1aca0a12e60d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:08.698519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb1f3e0-61be-4aef-b26a-1aca0a12e60d service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:08.707276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1195/2410] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:45:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:45:08.830690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:08.899314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f632bf35-3e25-43f9-9739-0a519d7c97af None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:08.903957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:08.904584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:08.904864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:08.905139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] PATH_INFO: `/v3/users/1b746e9d1eb44e12ac5b279825e1a01d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:08.906289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b746e9d1eb44e12ac5b279825e1a01d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:08.910376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:08.993198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b746e9d1eb44e12ac5b279825e1a01d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.003354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b746e9d1eb44e12ac5b279825e1a01d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.021737 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b746e9d1eb44e12ac5b279825e1a01d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.022505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b746e9d1eb44e12ac5b279825e1a01d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.034419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b746e9d1eb44e12ac5b279825e1a01d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.041582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1216/2411] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:08 2022] DELETE /identity/v3/users/1b746e9d1eb44e12ac5b279825e1a01d => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:09.046581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.047158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.047666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.091303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:09.092940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:09.170661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1196/2412] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:09.179520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:09.180823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:09.271985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0019ffbf-a866-40d3-aec5-6640be128de6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:09.278410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a1c074-98c2-4242-9cef-7e0202ecb0d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:09.279076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a1c074-98c2-4242-9cef-7e0202ecb0d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.279448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a1c074-98c2-4242-9cef-7e0202ecb0d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.279879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a1c074-98c2-4242-9cef-7e0202ecb0d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.291274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a1c074-98c2-4242-9cef-7e0202ecb0d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:09.293466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a1c074-98c2-4242-9cef-7e0202ecb0d6 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:09.305102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1217/2413] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:09.646060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:09.647578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:09.664972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90dcf67-86f0-4df5-a50c-395a4a04ec06 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:09.682665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:09.683440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.683725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.683997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] PATH_INFO: `/v3/projects/3b1c0bde41ce41858402312e47ab5d73` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.687855 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b1c0bde41ce41858402312e47ab5d73)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:09.689896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:09.752643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b1c0bde41ce41858402312e47ab5d73', 'request_id': 'req-6981feda-17ae-4d5d-884f-aa8f8110b506'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.760131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b1c0bde41ce41858402312e47ab5d73', 'request_id': 'req-6981feda-17ae-4d5d-884f-aa8f8110b506'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.767892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b1c0bde41ce41858402312e47ab5d73', 'request_id': 'req-6981feda-17ae-4d5d-884f-aa8f8110b506'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.779778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] The token cache is being invalidate because project 3b1c0bde41ce41858402312e47ab5d73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:09.780157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.780854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6981feda-17ae-4d5d-884f-aa8f8110b506 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:09.782399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1197/2414] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:09 2022] DELETE /identity/v3/projects/3b1c0bde41ce41858402312e47ab5d73 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:09.787409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.787906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.788024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.796979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:09.811775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:09.813268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:09.883916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1218/2415] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:09.892561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.892885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.893163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.914352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:09.915686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:09.927010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1198/2416] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:09.928285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:09.939731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd7150-4954-46c7-9569-1b705d8ea204 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:09.947643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:09.948241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.948595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.949501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.950166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:45:09.950971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:09.952198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:09.965375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1219/2417] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:45:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:45:09.969881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:09.979566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95a87e3-d385-418d-b780-0244eb31d3d6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:09.984846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:09.985531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:09.986091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:09.986368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:09.987059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:09.989134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:10.020856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1199/2418] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:09 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:10.022237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:10.033212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d3a01-25e4-4647-be09-04e6c4fa7a9a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:10.095773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:10.096462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:10.096822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:10.097188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:10.097923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:10.099911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:10.125040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1220/2419] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:10 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:10.130970 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:10.142658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2832eeff-89eb-4041-9cfb-192fd76fe5b1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:10.147680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:10.148292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:10.148573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:10.148849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:10.149666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:10.151354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:10.166391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1200/2420] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:10.167741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:10.178760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcea846-ed29-4356-808e-6ccce44302f5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:10.184409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dee7731-daa8-481a-a2d3-ca8af07b413b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:10.185202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dee7731-daa8-481a-a2d3-ca8af07b413b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:10.185489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dee7731-daa8-481a-a2d3-ca8af07b413b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:10.185764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dee7731-daa8-481a-a2d3-ca8af07b413b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:10.186416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dee7731-daa8-481a-a2d3-ca8af07b413b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:10.188050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dee7731-daa8-481a-a2d3-ca8af07b413b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:10.204798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1221/2421] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:10.205995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:10.217013 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe95681-cc5c-4555-b684-960a0435d4c1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:10.222343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:10.223279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:10.223577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:10.223849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] PATH_INFO: `/v3/projects/1461fc5c46f8433ebb5a9faa6d946b2f/users/c4c3d3df03944261acebdeac5d4154d7/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:10.242426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1461fc5c46f8433ebb5a9faa6d946b2f, user_id=c4c3d3df03944261acebdeac5d4154d7, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:10.245242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:10.248296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:10.269083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1201/2422] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:10 2022] PUT /identity/v3/projects/1461fc5c46f8433ebb5a9faa6d946b2f/users/c4c3d3df03944261acebdeac5d4154d7/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:10.280943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0cc117-1e5b-4279-a41f-6122b4859cdc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:10.281312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0cc117-1e5b-4279-a41f-6122b4859cdc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:10.281548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0cc117-1e5b-4279-a41f-6122b4859cdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:10.310770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7c0cc117-1e5b-4279-a41f-6122b4859cdc None None] MFA Rules not processed for user `c4c3d3df03944261acebdeac5d4154d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:10.312052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7c0cc117-1e5b-4279-a41f-6122b4859cdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:10.375930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:10.386849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1222/2423] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:10.395182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:10.397736 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:10.487048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b70b62-61a4-425d-bfc0-f3556d499e53 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:10.491435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:10.492159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:10.492517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:10.492870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:10.502059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:10.503645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:10.513181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1202/2424] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:11.218461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcc2e37-499a-4850-9188-521cc8f42ff0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:11.218981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcc2e37-499a-4850-9188-521cc8f42ff0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:11.219100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcc2e37-499a-4850-9188-521cc8f42ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:11.238612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9dcc2e37-499a-4850-9188-521cc8f42ff0 None None] MFA Rules not processed for user `2decd678421e4231bb3b58cd57d600bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:11.239856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9dcc2e37-499a-4850-9188-521cc8f42ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:11.295003 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1223/2425] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:11.853863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:11.855905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:11.912373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dac424-b0fd-4ae9-ae89-a90b6a133de2 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:11.915988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:11.916589 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:11.916864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:11.917133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:11.919502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:11.928933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:11.930389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:11.938600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1203/2426] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:11.953394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:11.953685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:11.953963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:11.973055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] MFA Rules not processed for user `c4c3d3df03944261acebdeac5d4154d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:11.974260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:11.981767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1224/2427] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:14.530260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:14.540996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd2d8b-6d1d-4a09-8067-46cb733f49d0 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:14.544834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:14.545524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:14.546056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:14.546698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:14.548814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:14.628516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:14.630001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:14.642098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1204/2428] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:19.256912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:19.258453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:19.306809 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:19.309493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:19.324496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc806f8-4f9b-4f8a-8d3d-59a8c6880328 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:19.329176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:19.329810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.330217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.330504 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] PATH_INFO: `/v3/users/bf68e214fd4740be96f2d6c4325f6ee9` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.331666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf68e214fd4740be96f2d6c4325f6ee9)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:19.333595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:19.337106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7100bf-8a16-42b1-8d91-9712ec89dc98 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:19.342433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:19.343170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.343476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.343788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.345498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:19.357865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:19.359645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:19.365240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:INSERT=1,keystone:SELECT=94,keystone:DELETE=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:19.373817 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1225/2429] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:19.388928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf68e214fd4740be96f2d6c4325f6ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.397341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf68e214fd4740be96f2d6c4325f6ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.410386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf68e214fd4740be96f2d6c4325f6ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.411089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf68e214fd4740be96f2d6c4325f6ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.417936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5575c3a6-9a67-48e0-b5f5-594e4f7fa6e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf68e214fd4740be96f2d6c4325f6ee9'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.423853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1205/2430] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:19 2022] DELETE /identity/v3/users/bf68e214fd4740be96f2d6c4325f6ee9 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:19.621490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:19.632557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c79c26-8eb3-4c7e-8405-1344b96a6c16 service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:19.693505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:19.694136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.694785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.694785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] PATH_INFO: `/v3/projects/7ad09c9b062641c19d3811d5c96a24f8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.695850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ad09c9b062641c19d3811d5c96a24f8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:19.697732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:19.733131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:19.750156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ad09c9b062641c19d3811d5c96a24f8', 'request_id': 'req-08667300-0361-4266-85fe-5ed09b60cd2b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.757297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ad09c9b062641c19d3811d5c96a24f8', 'request_id': 'req-08667300-0361-4266-85fe-5ed09b60cd2b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.763145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ad09c9b062641c19d3811d5c96a24f8', 'request_id': 'req-08667300-0361-4266-85fe-5ed09b60cd2b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.773962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] The token cache is being invalidate because project 7ad09c9b062641c19d3811d5c96a24f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:19.774255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.774881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08667300-0361-4266-85fe-5ed09b60cd2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:19.776564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1226/2431] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:19 2022] DELETE /identity/v3/projects/7ad09c9b062641c19d3811d5c96a24f8 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:19.781583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.781874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.782154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.801733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:19.802938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:19.865293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1206/2432] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:19.873773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.874094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.874379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.898894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:19.900155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:19.910325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1227/2433] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:19.914292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:19.927582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f6b2e2-36a8-4725-b6a1-c3db83aec831 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:19.933894 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:19.934614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.934988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.935332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.935997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:45:19.936305 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:19.937914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:19.950999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1207/2434] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:45:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:45:19.954579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:19.964938 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ff88d-cd41-4987-8c33-cb580f6f2ff6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:19.969904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:19.970666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:19.971088 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:19.971541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:19.972276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:19.974380 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:20.012066 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1228/2435] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:19 2022] POST /identity/v3/projects => generated 367 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:45:20.013275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:20.023485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2debd519-e92e-469e-b893-11d697c0d20d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:20.082790 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:20.083431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:20.083714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:20.083992 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:20.084627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:20.086661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:20.108459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1208/2436] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:20 2022] POST /identity/v3/users => generated 284 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:20.112708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:20.123577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab197c1-82ce-4e39-9777-35ea721321ad admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:20.128613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:20.129278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:20.129622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:20.129955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:20.130655 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:20.132335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:20.143521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1229/2437] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:20.147801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:20.158055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6aab8a-df31-4661-90eb-dec2a37617e3 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:20.162977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b165d-68bc-4981-9386-68e6ae2ef5d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:20.163605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b165d-68bc-4981-9386-68e6ae2ef5d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:20.163892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b165d-68bc-4981-9386-68e6ae2ef5d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:20.164160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b165d-68bc-4981-9386-68e6ae2ef5d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:20.164800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b165d-68bc-4981-9386-68e6ae2ef5d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:20.166723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b165d-68bc-4981-9386-68e6ae2ef5d6 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:20.177475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1209/2438] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:20.181786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:20.192850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b301af-c039-4de3-bcdb-0a2f297ceb94 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:20.197863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:20.198527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:20.199442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:20.199720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] PATH_INFO: `/v3/projects/93d9226d73ed43e9b6d2f0b1d2a27923/users/7ab5d8e0838c488cbd0f79850dbead22/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:20.215265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=93d9226d73ed43e9b6d2f0b1d2a27923, user_id=7ab5d8e0838c488cbd0f79850dbead22, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:20.217886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:20.231087 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1230/2439] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:20 2022] PUT /identity/v3/projects/93d9226d73ed43e9b6d2f0b1d2a27923/users/7ab5d8e0838c488cbd0f79850dbead22/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:20.242732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:20.242956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:20.243248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:20.265718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:20.274321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] MFA Rules not processed for user `7ab5d8e0838c488cbd0f79850dbead22`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:20.276191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:20.341705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1210/2440] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:20 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:20.352336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:20.354264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:20.415301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d40a3-9c77-4a1a-b4dc-ef2654022a2b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:20.419588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6e79e-8685-4299-8c56-b1942a80b327 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:20.420204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6e79e-8685-4299-8c56-b1942a80b327 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:20.420485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6e79e-8685-4299-8c56-b1942a80b327 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:20.420760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6e79e-8685-4299-8c56-b1942a80b327 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:20.429272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6e79e-8685-4299-8c56-b1942a80b327 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:20.430739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6e79e-8685-4299-8c56-b1942a80b327 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:20.440316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1231/2441] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:28.965358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:29.027857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ae2fff-cb8c-4e5c-b4c8-8467a29ee675 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:29.032385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:29.033008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.033283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.033550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] PATH_INFO: `/v3/users/7ab5d8e0838c488cbd0f79850dbead22` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.034657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ab5d8e0838c488cbd0f79850dbead22)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:29.036605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:29.085579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ab5d8e0838c488cbd0f79850dbead22'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.094329 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ab5d8e0838c488cbd0f79850dbead22'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.106859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ab5d8e0838c488cbd0f79850dbead22'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.107570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ab5d8e0838c488cbd0f79850dbead22'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.114628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ab5d8e0838c488cbd0f79850dbead22'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.120577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1211/2442] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:28 2022] DELETE /identity/v3/users/7ab5d8e0838c488cbd0f79850dbead22 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:29.125562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.126040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.126144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.147196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:29.148499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:29.199017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:29.216160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1232/2443] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:45:29.222767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:29.224205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:29.249618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:29.291020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88817b38-a24e-467e-ab1a-8f2d4ca4a4c0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:29.294831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3684f9-0eca-4961-afc0-d169a62f7555 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:29.296045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3684f9-0eca-4961-afc0-d169a62f7555 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.296322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3684f9-0eca-4961-afc0-d169a62f7555 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.296592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3684f9-0eca-4961-afc0-d169a62f7555 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.306343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3684f9-0eca-4961-afc0-d169a62f7555 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:29.307815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3684f9-0eca-4961-afc0-d169a62f7555 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:29.318860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1212/2444] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:29.643106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:29.644592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:29.657455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fc32e1-216b-472b-870d-e9ccda15ec60 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:29.663764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:29.664649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.664921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.665281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] PATH_INFO: `/v3/projects/93d9226d73ed43e9b6d2f0b1d2a27923` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.666515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=93d9226d73ed43e9b6d2f0b1d2a27923)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:29.668603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:29.737081 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93d9226d73ed43e9b6d2f0b1d2a27923', 'request_id': 'req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.745366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93d9226d73ed43e9b6d2f0b1d2a27923', 'request_id': 'req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.753733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93d9226d73ed43e9b6d2f0b1d2a27923', 'request_id': 'req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.764790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] The token cache is being invalidate because project 93d9226d73ed43e9b6d2f0b1d2a27923 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:29.765080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.765757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3a5e2101-1367-4db3-ad1f-cb3f2cf864ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:29.767596 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1233/2445] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:29 2022] DELETE /identity/v3/projects/93d9226d73ed43e9b6d2f0b1d2a27923 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:29.772407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.772766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.773130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.792639 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:29.793933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:29.853601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1213/2446] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:29.862206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.862515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.862792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.882164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:29.883323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:29.890984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1234/2447] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:29.895042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:29.905063 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b001d6-c028-42d3-b7e2-dd7b98d35de7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:29.909601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:29.910198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.910478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.910750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.911430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:45:29.911737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:29.913330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:29.925746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1214/2448] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:45:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:45:29.930143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:29.939517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea65ce3e-1067-4619-bf9a-5c9790446258 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:29.945209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:29.945809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:29.946092 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:29.946361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:29.947046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:29.949047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:29.981948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1235/2449] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:29 2022] POST /identity/v3/projects => generated 403 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:29.983143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:29.993178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d02207-73c6-4af3-affb-b8a13b53ba46 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.053681 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.054328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.054609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.054921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.055733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.057725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.084060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1215/2450] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:29 2022] POST /identity/v3/users => generated 302 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:30.085250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.096024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5449bf1-829d-4780-b8a3-446f8651ccf6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.101086 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.101689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.101970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.102243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.102885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.104500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.114527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1236/2451] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:30.118685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.128677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f9778d-ecb6-4603-a471-e7a735be2a94 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.134592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e6515-4f04-417f-866a-591f3fcf2929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.135312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e6515-4f04-417f-866a-591f3fcf2929 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.135601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e6515-4f04-417f-866a-591f3fcf2929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.135874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e6515-4f04-417f-866a-591f3fcf2929 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.136523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1e6515-4f04-417f-866a-591f3fcf2929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.138123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1e6515-4f04-417f-866a-591f3fcf2929 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.149772 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1216/2452] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:30.155336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.165487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b77640-1161-4d63-a044-a572dbc43448 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.170510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.171188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.171517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.171799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] PATH_INFO: `/v3/projects/b883fd615588445b807096a72ccbcbd8/users/1c0a8f672a694b96a2fef9213cad74ed/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.186394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b883fd615588445b807096a72ccbcbd8, user_id=1c0a8f672a694b96a2fef9213cad74ed, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.189038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.204640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1237/2453] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:30 2022] PUT /identity/v3/projects/b883fd615588445b807096a72ccbcbd8/users/1c0a8f672a694b96a2fef9213cad74ed/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:30.218045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.218337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.218579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.248191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] MFA Rules not processed for user `1c0a8f672a694b96a2fef9213cad74ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:30.249418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:30.273834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:30.325037 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1217/2454] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:30 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:30.330100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.400338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520c99c7-ee86-4d1c-bb9f-d0162d9d7ed7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.405106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.405716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.405999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.406284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.406923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.408866 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.429946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:30.442130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1238/2455] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:30 2022] POST /identity/v3/projects => generated 405 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:30.442828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.454506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0369ad-209c-46a8-b3e0-327cb2373bb7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.516241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.516862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.517143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.517423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.518061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.520054 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.547116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1218/2456] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:30 2022] POST /identity/v3/users => generated 309 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:45:30.548436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.559445 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8503173d-2831-4333-8394-4bf288dfdf65 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.564897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.565566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.565849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.566459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.567138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.568764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.580342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1239/2457] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:30.584470 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.594768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ff5c7e-d2e4-4772-a89a-a37ab8cd9f57 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.600184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.600735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.601020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.601295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] PATH_INFO: `/v3/projects/c8e19675c5ec4c7c97e2ed547384755c/users/05132e5220db4b7b89bba82788e349ad/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.618336 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8e19675c5ec4c7c97e2ed547384755c, user_id=05132e5220db4b7b89bba82788e349ad, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.621503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.647239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1219/2458] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:30 2022] PUT /identity/v3/projects/c8e19675c5ec4c7c97e2ed547384755c/users/05132e5220db4b7b89bba82788e349ad/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:30.648791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.661149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8960be08-5bb9-4c10-a473-617aef975173 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.743309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ade7ea-f676-443d-a75d-fe79ba67bbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.743934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ade7ea-f676-443d-a75d-fe79ba67bbc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.744218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ade7ea-f676-443d-a75d-fe79ba67bbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.744496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ade7ea-f676-443d-a75d-fe79ba67bbc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.745157 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ade7ea-f676-443d-a75d-fe79ba67bbc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.746779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ade7ea-f676-443d-a75d-fe79ba67bbc2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.761134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1240/2459] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:30 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:30.765471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.777695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c867b781-1f5d-4b5c-9992-2bfc2e78f1be admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.783324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.784005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.784469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.784810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] PATH_INFO: `/v3/projects/c8e19675c5ec4c7c97e2ed547384755c/users/05132e5220db4b7b89bba82788e349ad/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.786816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8e19675c5ec4c7c97e2ed547384755c, user_id=05132e5220db4b7b89bba82788e349ad, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:30.789509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:30.802964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1220/2460] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:30 2022] PUT /identity/v3/projects/c8e19675c5ec4c7c97e2ed547384755c/users/05132e5220db4b7b89bba82788e349ad/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:30.813897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.814187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.814467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:30.845701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] MFA Rules not processed for user `05132e5220db4b7b89bba82788e349ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:30.846911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:30.915820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1241/2461] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:30 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:30.927170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:30.928981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:30.986558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6447ea-b62a-4343-ac10-c4b4935d4bfe admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:30.990279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef81c1f-58fe-4685-ba0f-27d5d654d51f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:30.990887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef81c1f-58fe-4685-ba0f-27d5d654d51f service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:30.991204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef81c1f-58fe-4685-ba0f-27d5d654d51f service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:30.991483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef81c1f-58fe-4685-ba0f-27d5d654d51f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:31.001027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef81c1f-58fe-4685-ba0f-27d5d654d51f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:31.002452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef81c1f-58fe-4685-ba0f-27d5d654d51f service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:31.011680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1221/2462] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:30 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:31.159989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:31.223401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3044ad9f-c978-43ee-98d1-46ac39b35696 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:31.227956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:31.228586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:31.228922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:31.229206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] PATH_INFO: `/v3/users/1c0a8f672a694b96a2fef9213cad74ed` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:31.230326 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c0a8f672a694b96a2fef9213cad74ed)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:31.233716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:31.247784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:31.283695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c0a8f672a694b96a2fef9213cad74ed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.291064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c0a8f672a694b96a2fef9213cad74ed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.303947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c0a8f672a694b96a2fef9213cad74ed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.304599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c0a8f672a694b96a2fef9213cad74ed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.311605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c0a8f672a694b96a2fef9213cad74ed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.318863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1242/2463] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:31 2022] DELETE /identity/v3/users/1c0a8f672a694b96a2fef9213cad74ed => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:31.324068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:31.324353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:31.324624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:31.344710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:31.345913 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:31.354563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:31.411543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1222/2464] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:31.418024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:31.419494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:31.477370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c395d1ac-b9e8-436f-bcad-8ad25fa76e07 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:31.480944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:31.481543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:31.481821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:31.482111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:31.491186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:31.492598 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:31.501829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1243/2465] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:31.769283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:31.771192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:31.784506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ba6476-e875-4e01-9a02-951cab612a0a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:31.789174 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:31.789815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:31.790092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:31.790366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] PATH_INFO: `/v3/projects/b883fd615588445b807096a72ccbcbd8` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:31.791561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b883fd615588445b807096a72ccbcbd8)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:31.793476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:31.839874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b883fd615588445b807096a72ccbcbd8', 'request_id': 'req-84537600-1c10-4322-8421-9435afa7bc30'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.847131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b883fd615588445b807096a72ccbcbd8', 'request_id': 'req-84537600-1c10-4322-8421-9435afa7bc30'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.852954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b883fd615588445b807096a72ccbcbd8', 'request_id': 'req-84537600-1c10-4322-8421-9435afa7bc30'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.864473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] The token cache is being invalidate because project b883fd615588445b807096a72ccbcbd8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:31.864766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.865417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-84537600-1c10-4322-8421-9435afa7bc30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:31.866908 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1223/2466] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:31 2022] DELETE /identity/v3/projects/b883fd615588445b807096a72ccbcbd8 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:31.870898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:31.882215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30a6370-b075-44a1-93ad-a919b220d083 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:31.944735 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:31.945367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:31.945679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:31.945918 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] PATH_INFO: `/v3/users/05132e5220db4b7b89bba82788e349ad` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:31.947105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05132e5220db4b7b89bba82788e349ad)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:31.948998 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:31.988043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:32.000308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05132e5220db4b7b89bba82788e349ad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.007371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05132e5220db4b7b89bba82788e349ad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.020364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05132e5220db4b7b89bba82788e349ad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.021031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05132e5220db4b7b89bba82788e349ad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.022059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.022520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.022612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.028104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05132e5220db4b7b89bba82788e349ad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.033988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1244/2467] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:31 2022] DELETE /identity/v3/users/05132e5220db4b7b89bba82788e349ad => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:32.041981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:32.043229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.103665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1224/2468] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:32.128393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.129939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.208387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7364718-fcd9-462e-80a3-d231bfc3f499 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.213526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01321211-f6aa-48a4-b275-96ee40bb1fc0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.214139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01321211-f6aa-48a4-b275-96ee40bb1fc0 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.214425 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01321211-f6aa-48a4-b275-96ee40bb1fc0 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.214696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01321211-f6aa-48a4-b275-96ee40bb1fc0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.222956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01321211-f6aa-48a4-b275-96ee40bb1fc0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.224548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01321211-f6aa-48a4-b275-96ee40bb1fc0 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.233898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1245/2469] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:45:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 09 04:45:32.321930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.323573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.374828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:32.397973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ca1846-bdf8-4fa9-8aa0-5613b9d2b54c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.403157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.403782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.404086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.404389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] PATH_INFO: `/v3/projects/c8e19675c5ec4c7c97e2ed547384755c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.405544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8e19675c5ec4c7c97e2ed547384755c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.407488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.464862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8e19675c5ec4c7c97e2ed547384755c', 'request_id': 'req-695e23e4-5f35-4979-9c1e-ddbbb7632f79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.474598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8e19675c5ec4c7c97e2ed547384755c', 'request_id': 'req-695e23e4-5f35-4979-9c1e-ddbbb7632f79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.480636 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8e19675c5ec4c7c97e2ed547384755c', 'request_id': 'req-695e23e4-5f35-4979-9c1e-ddbbb7632f79'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.495731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] The token cache is being invalidate because project c8e19675c5ec4c7c97e2ed547384755c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:32.496106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.496775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-695e23e4-5f35-4979-9c1e-ddbbb7632f79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:32.498981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1225/2470] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:32 2022] DELETE /identity/v3/projects/c8e19675c5ec4c7c97e2ed547384755c => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:32.503859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.504155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.504429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.525684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:32.526907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.593407 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1246/2471] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:45:32.601485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.601775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.602052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.622859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:32.624150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.632333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1226/2472] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:32.636628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.646352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c5e37-d11f-4f4f-9b95-b66f05cede24 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.650959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.651574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.651853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.652122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.652763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:45:32.653062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.655170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.668656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1247/2473] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:45:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:45:32.673457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.682616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0806121-9722-4ed6-94e8-d3ce00becf92 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.687763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce499ac-1706-4d91-9602-9eb096ad4406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.688591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce499ac-1706-4d91-9602-9eb096ad4406 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.688893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce499ac-1706-4d91-9602-9eb096ad4406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.689231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce499ac-1706-4d91-9602-9eb096ad4406 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.689995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce499ac-1706-4d91-9602-9eb096ad4406 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.692299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce499ac-1706-4d91-9602-9eb096ad4406 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.725291 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1227/2474] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:32.727613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.740990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dc85ac-c943-4f55-aa59-2d23fb557256 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.774065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:32.798785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.799103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.799377 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.808370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.808970 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.809248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.809519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.810148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.812147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.818903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:32.820139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.834956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1248/2475] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/users => generated 282 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:32.838396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.848615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a5c4a0-714d-450e-866a-5d13c9f1068c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.853798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.854403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.854670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.854968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.855614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.858465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.869600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1249/2476] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:32.873525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.878635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1228/2477] 10.222.0.20 () {58 vars in 967 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:32.883371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd777bf-9cbc-4232-805f-b5dac6fa499f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.888374 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.889007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.889288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.889648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.890291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.890703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.891911 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.892117 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:32.906217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1250/2478] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:32.907198 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:32.916810 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e46056-5c07-41cf-8ea5-52bd89761bfb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.922061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e2dc3-079d-4866-99c4-2959d0c30cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:32.922692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e2dc3-079d-4866-99c4-2959d0c30cac admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.922996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e2dc3-079d-4866-99c4-2959d0c30cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.923279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e2dc3-079d-4866-99c4-2959d0c30cac admin admin] PATH_INFO: `/v3/projects/9b9a565b848644b38b7ae8632602dca7/users/0048fda6fbca424a90f82ac4bacaf1db/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.938944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e2dc3-079d-4866-99c4-2959d0c30cac admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b9a565b848644b38b7ae8632602dca7, user_id=0048fda6fbca424a90f82ac4bacaf1db, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:32.941886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e2dc3-079d-4866-99c4-2959d0c30cac admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:32.943176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:32.956854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1251/2479] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:32 2022] PUT /identity/v3/projects/9b9a565b848644b38b7ae8632602dca7/users/0048fda6fbca424a90f82ac4bacaf1db/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:32.968826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:32.969107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:32.969312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9a729e-7ef0-4358-9fd0-1229ff889bc8 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:32.969432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:32.996266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] MFA Rules not processed for user `0048fda6fbca424a90f82ac4bacaf1db`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:32.997485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:33.031176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.032031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.032031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.032275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.069973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1252/2480] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:32 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:45:33.074093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.091331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.093545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.101478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1229/2481] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:45:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:45:33.138306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca1702d-41c7-4a63-b24c-b140b14779a6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.143620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.144227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.144502 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.144774 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.145404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.147328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.153420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:33.177845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1253/2482] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:33 2022] POST /identity/v3/projects => generated 363 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:45:33.182648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.194056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017816c1-fc88-4950-b14a-2a316f3ef6f3 service swift] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.259021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.259668 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.260003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.260277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.260924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.262903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.289185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1230/2483] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:33 2022] POST /identity/v3/users => generated 288 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:33.293503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.306060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f7a44-d8c0-486c-ac64-a9396c5bf50d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.311304 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ffc1d1-35ed-4507-9c7f-dc3bfef79f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.311913 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ffc1d1-35ed-4507-9c7f-dc3bfef79f90 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.312193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ffc1d1-35ed-4507-9c7f-dc3bfef79f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.312462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ffc1d1-35ed-4507-9c7f-dc3bfef79f90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.313107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ffc1d1-35ed-4507-9c7f-dc3bfef79f90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.314706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ffc1d1-35ed-4507-9c7f-dc3bfef79f90 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.325619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1254/2484] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:33.331484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.342888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9662ede7-7497-4515-9c34-d2f316670ae1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.349466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.351243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.351243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.351487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] PATH_INFO: `/v3/projects/9c0c818349cb4bb8a91a9a31ea34bb75/users/ae5579036b2946b3a7607fa6e8814a45/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.357077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.357475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.357859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.377391 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c0c818349cb4bb8a91a9a31ea34bb75, user_id=ae5579036b2946b3a7607fa6e8814a45, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.380081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.380201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:33.381406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:33.389731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:33.395323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1231/2485] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:33 2022] PUT /identity/v3/projects/9c0c818349cb4bb8a91a9a31ea34bb75/users/ae5579036b2946b3a7607fa6e8814a45/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:33.399180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.442083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1255/2486] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:45:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:33.455301 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.464566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84530e9c-0beb-47ed-a284-fd6dfd40102f admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.469656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.470269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.470553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.470825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.471503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.473113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.483183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1232/2487] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:33 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:33.487197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.496982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf9dfaa-66e2-44cc-aff8-3866cd427d06 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.502257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d5edd8-0014-443b-a11c-7df5956b951b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.502886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5edd8-0014-443b-a11c-7df5956b951b admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.503203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5edd8-0014-443b-a11c-7df5956b951b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.503479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5edd8-0014-443b-a11c-7df5956b951b admin admin] PATH_INFO: `/v3/projects/9c0c818349cb4bb8a91a9a31ea34bb75/users/ae5579036b2946b3a7607fa6e8814a45/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.505434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5edd8-0014-443b-a11c-7df5956b951b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c0c818349cb4bb8a91a9a31ea34bb75, user_id=ae5579036b2946b3a7607fa6e8814a45, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.508064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5edd8-0014-443b-a11c-7df5956b951b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.518713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1233/2488] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:33 2022] PUT /identity/v3/projects/9c0c818349cb4bb8a91a9a31ea34bb75/users/ae5579036b2946b3a7607fa6e8814a45/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:33.523260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be688f1-1c64-4247-af0d-0fdb2d56e652 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.529066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.529386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.529667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.556638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] MFA Rules not processed for user `ae5579036b2946b3a7607fa6e8814a45`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:33.557848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:33.587813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f34c09f-c7ad-4cd7-859e-dc80711fda71 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.588416 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34c09f-c7ad-4cd7-859e-dc80711fda71 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.588695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34c09f-c7ad-4cd7-859e-dc80711fda71 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.588971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34c09f-c7ad-4cd7-859e-dc80711fda71 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.615123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:33.637509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1234/2489] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:33 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:45:33.649267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:33.650456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f34c09f-c7ad-4cd7-859e-dc80711fda71 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.650662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:33.652126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f34c09f-c7ad-4cd7-859e-dc80711fda71 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.663710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1256/2490] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:45:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 09 04:45:33.714159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b339a3-c92c-49bb-a7ee-585f081c70c3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:33.718116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:33.718732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:33.719043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:33.719317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:33.728600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:33.730034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:33.740838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1235/2491] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:33 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:45:34.021624 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:34.021979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:34.022360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:34.043311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:34.044644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:34.052705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1257/2492] 10.222.0.20 () {58 vars in 967 bytes} [Fri Sep 9 04:45:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:34.065583 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:34.078303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f38d0-a3af-401d-a74c-8722b1b8b4da service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:34.083554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb76a97-b042-404e-9040-7d39624bbab2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:34.084154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb76a97-b042-404e-9040-7d39624bbab2 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:34.084437 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb76a97-b042-404e-9040-7d39624bbab2 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:34.084708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb76a97-b042-404e-9040-7d39624bbab2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:34.090803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:34.094373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb76a97-b042-404e-9040-7d39624bbab2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:34.096304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb76a97-b042-404e-9040-7d39624bbab2 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:34.106192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1236/2493] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:45:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:45:34.173959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:34.253383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44813c7d-a3a3-4581-a20c-a3298760191f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:34.258579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:34.259341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:34.259685 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:34.260025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] PATH_INFO: `/v3/users/0048fda6fbca424a90f82ac4bacaf1db` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:34.261266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0048fda6fbca424a90f82ac4bacaf1db)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:34.263465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:34.314901 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0048fda6fbca424a90f82ac4bacaf1db'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.323851 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0048fda6fbca424a90f82ac4bacaf1db'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.338799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0048fda6fbca424a90f82ac4bacaf1db'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.339572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0048fda6fbca424a90f82ac4bacaf1db'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.348745 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0048fda6fbca424a90f82ac4bacaf1db'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.355277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1258/2494] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:34 2022] DELETE /identity/v3/users/0048fda6fbca424a90f82ac4bacaf1db => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:34.360110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:34.360549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:34.360853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:34.380510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:34.382896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:34.445386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1237/2495] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:34.456831 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:34.458482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:34.501757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:34.519316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfafdd6-500c-48e9-a024-7383be66b99c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:34.523187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:34.523693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:34.523976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:34.524256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:34.534237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:34.535716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:34.545218 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1259/2496] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:34.830439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:34.832096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:34.845346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff800c3-2abf-4b3d-8478-ebc2c8850ca0 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:34.862456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:34.863575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:34.864134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:34.864616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] PATH_INFO: `/v3/projects/9b9a565b848644b38b7ae8632602dca7` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:34.866123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b9a565b848644b38b7ae8632602dca7)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:34.872381 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:34.925123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b9a565b848644b38b7ae8632602dca7', 'request_id': 'req-2cac9da7-faea-4da3-9ba7-9b485f826389'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.934543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b9a565b848644b38b7ae8632602dca7', 'request_id': 'req-2cac9da7-faea-4da3-9ba7-9b485f826389'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.941513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b9a565b848644b38b7ae8632602dca7', 'request_id': 'req-2cac9da7-faea-4da3-9ba7-9b485f826389'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.953185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] The token cache is being invalidate because project 9b9a565b848644b38b7ae8632602dca7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:34.953480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.954163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:34.955813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1238/2497] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:34 2022] DELETE /identity/v3/projects/9b9a565b848644b38b7ae8632602dca7 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:34.963139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:34.969580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06facf87-2ab1-4185-94b3-9208b29d4eee service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.029902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.030542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.030824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.031147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] PATH_INFO: `/v3/users/ae5579036b2946b3a7607fa6e8814a45` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.032288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae5579036b2946b3a7607fa6e8814a45)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.034209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.086836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae5579036b2946b3a7607fa6e8814a45'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.094565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae5579036b2946b3a7607fa6e8814a45'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.112312 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae5579036b2946b3a7607fa6e8814a45'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.113012 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae5579036b2946b3a7607fa6e8814a45'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.120027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bd87d9ab-a782-4ad0-bf01-5cffc41c79cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae5579036b2946b3a7607fa6e8814a45'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.126422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1260/2498] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:34 2022] DELETE /identity/v3/users/ae5579036b2946b3a7607fa6e8814a45 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:35.404831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.406653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:35.420731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac9da7-faea-4da3-9ba7-9b485f826389 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.484590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.485439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.485755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.486069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] PATH_INFO: `/v3/projects/9c0c818349cb4bb8a91a9a31ea34bb75` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.487466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c0c818349cb4bb8a91a9a31ea34bb75)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.489379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.512682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:35.537457 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c0c818349cb4bb8a91a9a31ea34bb75', 'request_id': 'req-6916a82e-0881-4edb-b5b1-bf2171ab29dc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.544721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c0c818349cb4bb8a91a9a31ea34bb75', 'request_id': 'req-6916a82e-0881-4edb-b5b1-bf2171ab29dc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.550708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c0c818349cb4bb8a91a9a31ea34bb75', 'request_id': 'req-6916a82e-0881-4edb-b5b1-bf2171ab29dc'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.562803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] The token cache is being invalidate because project 9c0c818349cb4bb8a91a9a31ea34bb75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:35.563230 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.563936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-6916a82e-0881-4edb-b5b1-bf2171ab29dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:35.566534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1239/2499] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:35 2022] DELETE /identity/v3/projects/9c0c818349cb4bb8a91a9a31ea34bb75 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:35.572006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.572477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.572694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.592778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:35.593919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:35.647177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:35.657923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1261/2500] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:35.665779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.666098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.666375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.695652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:35.696886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:35.704858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1240/2501] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:45:35.709154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.719561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b458e27a-d35d-423d-8205-645b1442a8c7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.724138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.724736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.725019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.725297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.725946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:45:35.726513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.728156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.740279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1262/2502] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:45:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:45:35.744480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.753599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c25c6d8-32d6-4456-9bcf-d9ddfa71da60 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.758253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.758848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.759167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.759443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.760180 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.762308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.789249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1241/2503] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:35 2022] POST /identity/v3/projects => generated 377 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:45:35.793707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.804284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa52473-380d-487b-860a-957150c78295 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.860376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.860904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.861180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.861489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.862079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.864000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.884327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1263/2504] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:35 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:35.888675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.898740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948f84a-4fa7-4893-8a1c-b5504e7e7591 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.903288 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.903883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.904161 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.904923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.905559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.907166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.916755 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1242/2505] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:35.921232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.931592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db7527-3be9-415e-87be-b08c75754491 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.936248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb6e8b9-8f12-4f55-93f5-c330b626597e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.936834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb6e8b9-8f12-4f55-93f5-c330b626597e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.937118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb6e8b9-8f12-4f55-93f5-c330b626597e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.937403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb6e8b9-8f12-4f55-93f5-c330b626597e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.938043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb6e8b9-8f12-4f55-93f5-c330b626597e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.939707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb6e8b9-8f12-4f55-93f5-c330b626597e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:35.950323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1264/2506] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:45:35.954865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:35.964596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6467fdf-12fa-497e-968e-dcf28e1813bc admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:35.969856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:35.970488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:35.970768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:35.971082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] PATH_INFO: `/v3/projects/7bcf9798b7ed4bea9c8d50c7ccdeac2c/users/1beb620815ef435b89f37fcda1b07c5c/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:35.985181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bcf9798b7ed4bea9c8d50c7ccdeac2c, user_id=1beb620815ef435b89f37fcda1b07c5c, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:35.987839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.000155 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1243/2507] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:35 2022] PUT /identity/v3/projects/7bcf9798b7ed4bea9c8d50c7ccdeac2c/users/1beb620815ef435b89f37fcda1b07c5c/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:36.011193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.011483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.011761 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.040449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] MFA Rules not processed for user `1beb620815ef435b89f37fcda1b07c5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:36.041662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.105882 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1265/2508] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:36.110442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.171475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f8c72-1dfd-4b41-8a89-cfef5eb46484 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.176176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.176759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.177042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.177320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.177995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.179947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.206392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1244/2509] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:45:36 2022] POST /identity/v3/projects => generated 375 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:36.215546 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.227173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7282ef3d-9e71-4f1c-9a1d-84b2838c8c4c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.245763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.247484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.285646 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:36.286707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:36.294583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.295457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.296055 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.296225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.297521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.300103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.319648 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb9c293-6a2f-4402-ac82-bb58098437ad admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.322879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1266/2510] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:45:36 2022] POST /identity/v3/users => generated 294 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:45:36.325215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.325850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.326615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.326884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] PATH_INFO: `/v3/users/c4c3d3df03944261acebdeac5d4154d7` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.327256 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.328051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4c3d3df03944261acebdeac5d4154d7)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.329955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.337545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb12789-c302-48ec-b1fd-7475e747121f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.342752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.343394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.343673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.343948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.344583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.346147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.356101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1267/2511] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:36.360243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.370213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d74633-a39b-4630-a368-052823c0498f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.376209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.376857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.377305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.377589 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] PATH_INFO: `/v3/projects/54ca422d61e0476cb5b7452336774f4c/users/306c152dae21425d8b4e1077bc768b52/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.382719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4c3d3df03944261acebdeac5d4154d7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.391106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4c3d3df03944261acebdeac5d4154d7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.394385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54ca422d61e0476cb5b7452336774f4c, user_id=306c152dae21425d8b4e1077bc768b52, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.397051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.403141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4c3d3df03944261acebdeac5d4154d7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.403814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4c3d3df03944261acebdeac5d4154d7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.411722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1268/2512] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:36 2022] PUT /identity/v3/projects/54ca422d61e0476cb5b7452336774f4c/users/306c152dae21425d8b4e1077bc768b52/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:36.412651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-625ca831-bf9e-4030-9346-e8069b85afe2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4c3d3df03944261acebdeac5d4154d7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.415732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.418097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.419321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1245/2513] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:45:36 2022] DELETE /identity/v3/users/c4c3d3df03944261acebdeac5d4154d7 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:36.423586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.423873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.424364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.444570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:36.445900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.488070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e5907d-ec3a-46a0-ba95-25846c04dff0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.512485 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1246/2514] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:45:36.519791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.521242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.548922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.549698 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.549986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.550265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.550912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.552557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.567530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1269/2515] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:45:36 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:45:36.568843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.579264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b45d9-1536-48c9-a830-ce77b2395086 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.580632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70e3b36-bb7d-4122-97af-662de30f8a3f None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.584113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15f6e21-0a92-4b64-93bf-7af908943492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.584574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.584739 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15f6e21-0a92-4b64-93bf-7af908943492 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.585020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15f6e21-0a92-4b64-93bf-7af908943492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.585177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.585282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15f6e21-0a92-4b64-93bf-7af908943492 admin admin] PATH_INFO: `/v3/projects/54ca422d61e0476cb5b7452336774f4c/users/306c152dae21425d8b4e1077bc768b52/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.585456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.585727 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.587253 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15f6e21-0a92-4b64-93bf-7af908943492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54ca422d61e0476cb5b7452336774f4c, user_id=306c152dae21425d8b4e1077bc768b52, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.589987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15f6e21-0a92-4b64-93bf-7af908943492 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.591911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:36.595386 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.597357 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.601014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1270/2516] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:45:36 2022] PUT /identity/v3/projects/54ca422d61e0476cb5b7452336774f4c/users/306c152dae21425d8b4e1077bc768b52/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:45:36.611600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.611888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.612167 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.615276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1247/2517] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:36.643594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] MFA Rules not processed for user `306c152dae21425d8b4e1077bc768b52`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:45:36.644883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.676221 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:36.722713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1271/2518] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:45:36 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:45:36.734297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.735921 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.784759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:36.786508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.800758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ed69a4-0950-4c84-96b2-838dadbba39c service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.855087 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.855695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.855974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.856251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.856597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75efc398-39e8-4365-b512-c31c11b1d537 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:36.858543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:36.861760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:36.862387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:36.862667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:36.862947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] PATH_INFO: `/v3/projects/1461fc5c46f8433ebb5a9faa6d946b2f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:36.864115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1461fc5c46f8433ebb5a9faa6d946b2f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.866048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.914284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1461fc5c46f8433ebb5a9faa6d946b2f', 'request_id': 'req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.917932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:36.920330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:36.922994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1461fc5c46f8433ebb5a9faa6d946b2f', 'request_id': 'req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.929252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1461fc5c46f8433ebb5a9faa6d946b2f', 'request_id': 'req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.929935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1248/2519] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:36.941512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] The token cache is being invalidate because project 1461fc5c46f8433ebb5a9faa6d946b2f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:45:36.941803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.942474 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:45:36.944212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1272/2520] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:45:36 2022] DELETE /identity/v3/projects/1461fc5c46f8433ebb5a9faa6d946b2f => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:45:41.576496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:41.585031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa018de-f83a-4af9-8054-7200156708b6 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:41.632543 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:41.633136 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:41.633413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:41.633684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:41.635546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:41.656361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:41.707392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:41.708826 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:41.718767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1249/2521] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:41 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:45:46.938290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=67,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:51.714490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=21 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:45:57.442684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:45:57.444118 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:57.501939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eba9e0-5e01-4bf9-9abc-29a2bd1d8d91 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:45:57.505566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:45:57.506168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:45:57.506449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:45:57.506721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:45:57.508457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:45:57.571785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:45:57.573628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:45:57.583847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1273/2522] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:45:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:00.789916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:00.791493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:00.807957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5c2f26-45cf-4f6f-9569-61be859f5848 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:00.812595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:00.813232 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:00.813516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:00.813794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] PATH_INFO: `/v3/users/befee01f638c4e7681b30c64acad6a6a` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:00.814915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=befee01f638c4e7681b30c64acad6a6a)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:00.817168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:00.863312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'befee01f638c4e7681b30c64acad6a6a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:00.874347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'befee01f638c4e7681b30c64acad6a6a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:00.886343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'befee01f638c4e7681b30c64acad6a6a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:00.887049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'befee01f638c4e7681b30c64acad6a6a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:00.894258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-54f714dd-aaf7-461b-81e1-2e67333bcb35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'befee01f638c4e7681b30c64acad6a6a'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:00.900083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1250/2523] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:00 2022] DELETE /identity/v3/users/befee01f638c4e7681b30c64acad6a6a => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:01.071733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.085150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb91993-7590-449f-ba26-bd2c7cb1c52c service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.146067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.146850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.147089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.147486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] PATH_INFO: `/v3/projects/5176b458a4a8481eaa37d8693b51f15e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.148483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5176b458a4a8481eaa37d8693b51f15e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.150547 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.203581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5176b458a4a8481eaa37d8693b51f15e', 'request_id': 'req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:01.206299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:01.212314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5176b458a4a8481eaa37d8693b51f15e', 'request_id': 'req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:01.220194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5176b458a4a8481eaa37d8693b51f15e', 'request_id': 'req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:01.232062 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] The token cache is being invalidate because project 5176b458a4a8481eaa37d8693b51f15e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:01.232354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:01.233007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6a69567-6f1d-44b4-881d-f85a91fbf4d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:01.234721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1274/2524] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:01 2022] DELETE /identity/v3/projects/5176b458a4a8481eaa37d8693b51f15e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:01.239552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.239873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.240149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.260202 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:01.261339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:01.320986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1251/2525] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:01.329705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.329993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.330270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.349461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:01.350654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:01.358800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1275/2526] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:01.362882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.372945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9571019b-e68c-4501-92b6-24ab3a7e854c None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.377920 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.378605 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.378888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.379222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.379977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:01.380214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.381825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.393923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1252/2527] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:01.398129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.407389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e7fc76-af97-4278-ac8a-1794d70017f0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.412017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.412610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.412892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.413166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.413824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.415794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.442543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1276/2528] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/projects => generated 391 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:01.446811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.457393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c6bb91-3158-4c9c-8948-c51c906f95b1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.513304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.513831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.514109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.514383 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.515034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.516927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.524741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:01.541941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1253/2529] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/users => generated 296 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:01.543268 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.553854 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee27349-b620-4c91-9639-44c5b2849f6e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.558455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.559089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.559371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.560280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.560923 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.562509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.572520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1277/2530] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:01.576298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.586282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9665a8cd-c4ab-4939-843c-0ab810063881 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.591106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d84f07-1201-4529-9af3-1027c89c6a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.591696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d84f07-1201-4529-9af3-1027c89c6a6f admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.591976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d84f07-1201-4529-9af3-1027c89c6a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.592250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d84f07-1201-4529-9af3-1027c89c6a6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.592924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d84f07-1201-4529-9af3-1027c89c6a6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.594510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d84f07-1201-4529-9af3-1027c89c6a6f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.604912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1254/2531] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:01.609487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.619274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6597809-2c27-474a-9ec5-39c24b73d54f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.624435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.625069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.625349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.625622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] PATH_INFO: `/v3/projects/ffb24d32ec75468fa570374a28101257/users/0bc4cc88bf0a4c20a32367096c0248e6/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.640906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffb24d32ec75468fa570374a28101257, user_id=0bc4cc88bf0a4c20a32367096c0248e6, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.643563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.656206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1278/2532] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:01 2022] PUT /identity/v3/projects/ffb24d32ec75468fa570374a28101257/users/0bc4cc88bf0a4c20a32367096c0248e6/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:01.667734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.668027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.668303 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.696895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] MFA Rules not processed for user `0bc4cc88bf0a4c20a32367096c0248e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:01.698140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:01.764159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1255/2533] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:01.768625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.836695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af53ef-3d37-4c2d-9fbe-10320dda48ba admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.841844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.842435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.842711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.842996 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.843629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.845493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.872010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1279/2534] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/projects => generated 393 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:01.876455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.887637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89867fad-4e70-4c94-96e9-2a20765930bb None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.945611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.946224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.946507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.946782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.947460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.949375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:01.969767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1256/2535] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:01 2022] POST /identity/v3/users => generated 303 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:01.973827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:01.985099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46326292-7a13-42d4-ad61-b3ab49ef8563 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:01.989695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:01.990274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:01.990566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:01.990841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:01.991516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:01.993115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:02.003522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1280/2536] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:02.007394 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:02.019227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dbe643-4860-4f28-a859-b5f9d47ca06d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:02.023742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:02.024400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:02.024665 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:02.024937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] PATH_INFO: `/v3/projects/540448c889be4f02830d5803bdaf604d/users/0f1f1b3ab34243b1a15d4bebc6f9e566/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:02.039804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] RBAC: Authorizing `identity:create_grant(project_id=540448c889be4f02830d5803bdaf604d, user_id=0f1f1b3ab34243b1a15d4bebc6f9e566, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:02.042434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:02.055546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1257/2537] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:02 2022] PUT /identity/v3/projects/540448c889be4f02830d5803bdaf604d/users/0f1f1b3ab34243b1a15d4bebc6f9e566/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:02.059420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:02.069609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140258a7-43b6-46ec-8f7c-4beb6738a3f7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:02.116403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:02.127488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4888fee5-27af-489e-9708-198768becd10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:02.128084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4888fee5-27af-489e-9708-198768becd10 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:02.128365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4888fee5-27af-489e-9708-198768becd10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:02.128640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4888fee5-27af-489e-9708-198768becd10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:02.129272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4888fee5-27af-489e-9708-198768becd10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:02.130841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4888fee5-27af-489e-9708-198768becd10 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:02.140457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1281/2538] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:02 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:02.144563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:02.155221 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed30d10d-b229-4473-95f6-f764bc6ac866 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:02.159776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:02.160428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:02.160710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:02.160988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] PATH_INFO: `/v3/projects/540448c889be4f02830d5803bdaf604d/users/0f1f1b3ab34243b1a15d4bebc6f9e566/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:02.162912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] RBAC: Authorizing `identity:create_grant(project_id=540448c889be4f02830d5803bdaf604d, user_id=0f1f1b3ab34243b1a15d4bebc6f9e566, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:02.165566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:02.176243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1258/2539] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:02 2022] PUT /identity/v3/projects/540448c889be4f02830d5803bdaf604d/users/0f1f1b3ab34243b1a15d4bebc6f9e566/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:02.187628 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:02.187919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:02.188194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:02.217111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] MFA Rules not processed for user `0f1f1b3ab34243b1a15d4bebc6f9e566`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:02.218338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:02.285538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1282/2540] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:02 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:02.296315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:02.298084 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:02.320751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:02.356998 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0037805e-5707-4e07-a1e3-727f5d46c829 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:02.360608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:02.361198 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:02.361476 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:02.361751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:02.370447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:02.371890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:02.381318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1259/2541] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:02 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:02.429707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:02.431133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:02.443135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb258c-4ebe-4ae8-99c6-aa96bb13fef9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:02.446680 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbeaba3-d65d-4016-9eaf-4d06f9929989 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:02.447314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbeaba3-d65d-4016-9eaf-4d06f9929989 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:02.447592 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbeaba3-d65d-4016-9eaf-4d06f9929989 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:02.447863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbeaba3-d65d-4016-9eaf-4d06f9929989 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:02.503115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbeaba3-d65d-4016-9eaf-4d06f9929989 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:02.504525 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbeaba3-d65d-4016-9eaf-4d06f9929989 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:02.513700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1283/2542] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:02 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:05.797728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:05.863014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06736820-bf86-4e86-99e7-c3fcdcc11d00 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:05.867588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:05.868328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:05.868623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:05.868899 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] PATH_INFO: `/v3/users/0bc4cc88bf0a4c20a32367096c0248e6` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:05.870019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0bc4cc88bf0a4c20a32367096c0248e6)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:05.871977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:05.920146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0bc4cc88bf0a4c20a32367096c0248e6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:05.927588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0bc4cc88bf0a4c20a32367096c0248e6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:05.941052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0bc4cc88bf0a4c20a32367096c0248e6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:05.942074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0bc4cc88bf0a4c20a32367096c0248e6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:05.949194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0bc4cc88bf0a4c20a32367096c0248e6'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:05.955897 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1260/2543] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:05 2022] DELETE /identity/v3/users/0bc4cc88bf0a4c20a32367096c0248e6 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:05.960091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:05.960549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:05.960671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:05.979896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:05.981108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.039344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1284/2544] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:06.049403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:06.050807 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.096324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:06.109682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c24c8-842e-44c6-82ca-87672a83d830 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:06.113535 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:06.114148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.114539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.114819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.124595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:06.126024 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:06.135395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1261/2545] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:06.397487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:06.398969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.412482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71af0c0-368a-4fd2-a92e-05118c0bda3b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:06.417096 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:06.417750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.418027 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.418306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] PATH_INFO: `/v3/projects/ffb24d32ec75468fa570374a28101257` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.419455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffb24d32ec75468fa570374a28101257)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:06.421455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:06.428219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:06.470270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffb24d32ec75468fa570374a28101257', 'request_id': 'req-5ea83343-fd5f-4a88-96c8-97a96a652d09'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.477457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffb24d32ec75468fa570374a28101257', 'request_id': 'req-5ea83343-fd5f-4a88-96c8-97a96a652d09'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.483190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffb24d32ec75468fa570374a28101257', 'request_id': 'req-5ea83343-fd5f-4a88-96c8-97a96a652d09'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.494310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] The token cache is being invalidate because project ffb24d32ec75468fa570374a28101257 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:06.494600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.495243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.496729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1285/2546] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:06 2022] DELETE /identity/v3/projects/ffb24d32ec75468fa570374a28101257 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:06.501317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:06.510730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29101586-4140-4023-9b52-02b92a19f2cd service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:06.567240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:06.567860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.568140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.568417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] PATH_INFO: `/v3/users/0f1f1b3ab34243b1a15d4bebc6f9e566` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.569533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f1f1b3ab34243b1a15d4bebc6f9e566)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:06.571490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:06.615922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f1f1b3ab34243b1a15d4bebc6f9e566'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.624334 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f1f1b3ab34243b1a15d4bebc6f9e566'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.624714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:06.626131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.636971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f1f1b3ab34243b1a15d4bebc6f9e566'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.637718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f1f1b3ab34243b1a15d4bebc6f9e566'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.639602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea83343-fd5f-4a88-96c8-97a96a652d09 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:06.644480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:06.645103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.645223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c50f2cfa-b538-4fc3-8a20-8a2450ac0221 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f1f1b3ab34243b1a15d4bebc6f9e566'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.645386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.645660 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] PATH_INFO: `/v3/users/1beb620815ef435b89f37fcda1b07c5c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.646784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1beb620815ef435b89f37fcda1b07c5c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:06.648727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:06.651595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1262/2547] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:06 2022] DELETE /identity/v3/users/0f1f1b3ab34243b1a15d4bebc6f9e566 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:06.703672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1beb620815ef435b89f37fcda1b07c5c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.714120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1beb620815ef435b89f37fcda1b07c5c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.729424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1beb620815ef435b89f37fcda1b07c5c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.730402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1beb620815ef435b89f37fcda1b07c5c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.739376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1beb620815ef435b89f37fcda1b07c5c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:06.745754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1286/2548] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:06 2022] DELETE /identity/v3/users/1beb620815ef435b89f37fcda1b07c5c => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:06.751034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.751322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.751596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.772181 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:06.773514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.837775 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1263/2549] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:06.848028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:06.849468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.908074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd9fcc-5bbc-4449-9d6f-b87bfb3b5f00 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:06.912278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c1e75c-5a59-4804-844b-00203073aef2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:06.912949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c1e75c-5a59-4804-844b-00203073aef2 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.913262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c1e75c-5a59-4804-844b-00203073aef2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.913581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c1e75c-5a59-4804-844b-00203073aef2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.925230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c1e75c-5a59-4804-844b-00203073aef2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:06.926652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c1e75c-5a59-4804-844b-00203073aef2 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:06.931604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:06.933604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:06.938341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1287/2550] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:06.942689 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:06.946460 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaee4af-2c93-4a9f-bcce-85c8382fa9a4 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:06.958222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:06.959019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:06.959368 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:06.961431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] PATH_INFO: `/v3/projects/540448c889be4f02830d5803bdaf604d` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:06.962774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] RBAC: Authorizing `identity:delete_project(project_id=540448c889be4f02830d5803bdaf604d)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:06.964871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.017277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '540448c889be4f02830d5803bdaf604d', 'request_id': 'req-c38cb103-f190-426e-bdf5-5f487e583008'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.024500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '540448c889be4f02830d5803bdaf604d', 'request_id': 'req-c38cb103-f190-426e-bdf5-5f487e583008'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.030370 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '540448c889be4f02830d5803bdaf604d', 'request_id': 'req-c38cb103-f190-426e-bdf5-5f487e583008'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.041455 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] The token cache is being invalidate because project 540448c889be4f02830d5803bdaf604d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:07.041746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.042407 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-c38cb103-f190-426e-bdf5-5f487e583008 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.044538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1264/2551] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:06 2022] DELETE /identity/v3/projects/540448c889be4f02830d5803bdaf604d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:07.050125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.050627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.050727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.072371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:07.073590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.078024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:07.139768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1288/2552] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:07.147557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.147881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.148160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.168745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:07.170529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.178878 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1265/2553] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:07.182895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.192730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8ea73-e468-4511-ab36-c85da405e4bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.197386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.197985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.198277 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.198554 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.199235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:07.199543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.201173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.213780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1289/2554] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:07.219466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.228671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24485329-3626-4b0f-b6ae-44db8cdc59c6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.230781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.233272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.239540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.240158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.240488 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.240765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.241400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.243406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.253138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a84064-4141-4807-b977-04bf02b74a49 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.265287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.266054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.266341 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.266599 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] PATH_INFO: `/v3/projects/7bcf9798b7ed4bea9c8d50c7ccdeac2c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.267791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bcf9798b7ed4bea9c8d50c7ccdeac2c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.270295 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.276915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1266/2555] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/projects => generated 379 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:07.281423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.319188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bcf9798b7ed4bea9c8d50c7ccdeac2c', 'request_id': 'req-7c7be451-294e-40a1-acf9-9750daf99a63'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.326900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bcf9798b7ed4bea9c8d50c7ccdeac2c', 'request_id': 'req-7c7be451-294e-40a1-acf9-9750daf99a63'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.332773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bcf9798b7ed4bea9c8d50c7ccdeac2c', 'request_id': 'req-7c7be451-294e-40a1-acf9-9750daf99a63'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.344797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] The token cache is being invalidate because project 7bcf9798b7ed4bea9c8d50c7ccdeac2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:07.345083 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.345731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.347254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1290/2556] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:07 2022] DELETE /identity/v3/projects/7bcf9798b7ed4bea9c8d50c7ccdeac2c => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:07.350993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.351367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3f6f1a-592b-4c5d-9a6a-2189f9e43ead admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.352275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.409059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.409671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.409951 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.410231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.410860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.412808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.416777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7be451-294e-40a1-acf9-9750daf99a63 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.421930 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.422569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.422825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.423176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] PATH_INFO: `/v3/users/306c152dae21425d8b4e1077bc768b52` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.424641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:07.424784 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] RBAC: Authorizing `identity:delete_user(user_id=306c152dae21425d8b4e1077bc768b52)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.427405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.440090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1267/2557] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/users => generated 290 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:07.441139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.443048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.452375 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:07.455300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438f8241-47f0-414e-80be-f86685fd067a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.460494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.460905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.461185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.461458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.462091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.463735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.474398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1268/2558] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:07.478036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.480794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '306c152dae21425d8b4e1077bc768b52'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.488512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae8922a-7344-46a5-893b-86dec86b1342 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.489608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '306c152dae21425d8b4e1077bc768b52'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.493852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.494445 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.494730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.495046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.495718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.497373 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.502393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '306c152dae21425d8b4e1077bc768b52'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.503105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '306c152dae21425d8b4e1077bc768b52'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.508000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1269/2559] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:07.512028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.514275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ac091730-ee9d-4f6f-952a-52d92c393228 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '306c152dae21425d8b4e1077bc768b52'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.514518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.521448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1291/2560] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:07 2022] DELETE /identity/v3/users/306c152dae21425d8b4e1077bc768b52 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:07.527797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd06a18c-194d-47b3-bff7-d8ab8e4768fc admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.588255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.589090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.589375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.589656 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] PATH_INFO: `/v3/projects/21c35f3c9b6149abaa8d7cb44566db18/users/4c99ad129b1a4ec19867e54370ec8c30/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.609489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=21c35f3c9b6149abaa8d7cb44566db18, user_id=4c99ad129b1a4ec19867e54370ec8c30, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.612546 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.626406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1270/2561] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:07 2022] PUT /identity/v3/projects/21c35f3c9b6149abaa8d7cb44566db18/users/4c99ad129b1a4ec19867e54370ec8c30/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:07.637104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.637396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.638193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.667158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] MFA Rules not processed for user `4c99ad129b1a4ec19867e54370ec8c30`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:07.668417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.736266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1292/2562] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:07.740972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.813770 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.815262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.817955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63ea21-1149-4065-836d-3a47408f9e9f admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.823116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.824040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.824327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.824600 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.825729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.828816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.831056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b6269b-7d0f-4135-9aa2-a35033b1b7f3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.837020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.837765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.838164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.838540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] PATH_INFO: `/v3/projects/54ca422d61e0476cb5b7452336774f4c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.839834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] RBAC: Authorizing `identity:delete_project(project_id=54ca422d61e0476cb5b7452336774f4c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.842489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:07.861650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1271/2563] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/projects => generated 377 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:07.862718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:07.894555 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54ca422d61e0476cb5b7452336774f4c', 'request_id': 'req-9168b5d3-09e7-4583-9a5f-f863aeafb745'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.902405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54ca422d61e0476cb5b7452336774f4c', 'request_id': 'req-9168b5d3-09e7-4583-9a5f-f863aeafb745'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.908507 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54ca422d61e0476cb5b7452336774f4c', 'request_id': 'req-9168b5d3-09e7-4583-9a5f-f863aeafb745'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.909928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:07.920643 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] The token cache is being invalidate because project 54ca422d61e0476cb5b7452336774f4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:07.920953 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.921654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9168b5d3-09e7-4583-9a5f-f863aeafb745 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:07.923173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1293/2564] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:07 2022] DELETE /identity/v3/projects/54ca422d61e0476cb5b7452336774f4c => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:07.927764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961af813-4a36-4194-8b6c-e3100a7c8139 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.928079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961af813-4a36-4194-8b6c-e3100a7c8139 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.928354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961af813-4a36-4194-8b6c-e3100a7c8139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.931578 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cdb4db-e94c-47aa-a067-f8e0f4d57b31 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:07.952279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-961af813-4a36-4194-8b6c-e3100a7c8139 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:07.953824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-961af813-4a36-4194-8b6c-e3100a7c8139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:07.960518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:07.989666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:07.990298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:07.990611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:07.990887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:07.991556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:07.993503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.015073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1272/2565] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/users => generated 289 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:08.017322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1294/2566] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:08.019231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.020847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:08.025065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.025356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.025634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.032320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340220b3-0f74-461f-a50a-e8c51d09ea2d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.036871 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.037510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.037791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.038061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.038704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.040297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.045264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:08.046432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:08.050521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1273/2567] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:08.054432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.054575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1295/2568] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:08.058230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.064844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a6b945-c64f-409c-8c4e-ed5b9fcdbf5a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.068174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a2ec72-42e5-4155-9473-8a89cfab4d74 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.069778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.070435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.070718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.071028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.071669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.072992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.073242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.073572 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.073852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.074174 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.074838 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:08.075777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.077342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.084369 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1274/2569] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:08.088452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.089228 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1296/2570] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:08.093381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.098734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ca0bf0-42bf-480e-8a2b-768cc92c29c1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.103219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1417586a-d796-40a2-8497-dbaabf879473 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.103603 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8a02f6-6db9-407a-bc7e-5ffe7a4501d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.104247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8a02f6-6db9-407a-bc7e-5ffe7a4501d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.104544 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8a02f6-6db9-407a-bc7e-5ffe7a4501d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.104805 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8a02f6-6db9-407a-bc7e-5ffe7a4501d0 admin admin] PATH_INFO: `/v3/projects/adb69ccfbc7f4dc2bcf109bc535860b8/users/f7743fd355ae43c9ae6d68cd799faeca/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.112343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.112950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.113257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.113534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.114170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.116111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.119410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8a02f6-6db9-407a-bc7e-5ffe7a4501d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adb69ccfbc7f4dc2bcf109bc535860b8, user_id=f7743fd355ae43c9ae6d68cd799faeca, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.122538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8a02f6-6db9-407a-bc7e-5ffe7a4501d0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.135190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1275/2571] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:08 2022] PUT /identity/v3/projects/adb69ccfbc7f4dc2bcf109bc535860b8/users/f7743fd355ae43c9ae6d68cd799faeca/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:08.149528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1297/2572] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/projects => generated 389 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:08.149707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.149707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.149707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.150462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.175955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] MFA Rules not processed for user `f7743fd355ae43c9ae6d68cd799faeca`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:08.177169 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:08.216941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e1564-ac63-4f68-9765-4a4ca0155064 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.221521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.222119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.222396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.222665 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.223316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.225222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.247733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1276/2573] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:08.250960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1298/2574] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/users => generated 295 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:08.252134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.252281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.263238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de16e7d-17d3-48c1-b996-fde4a8376a89 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.263420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf039c3d-233a-443c-b8b7-3f142270aa2a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.268071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.268361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.268671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.268950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.269201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.269352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.269487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.269766 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.269865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.270420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.271764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.272038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.282664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1277/2575] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:08.287034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.298452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e995bf-736d-4542-98ce-0b46b3f17650 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.301870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1299/2576] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:08.303058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.303645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.303925 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.304194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.304831 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.305862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.306411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.312521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:08.322308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1278/2577] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:08.323217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.335490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75017fa-9fb3-47f4-88f0-c7469019eae2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.352638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:08.376725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66e9ebe-4926-4642-b90f-113172aecf5d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.381602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.382216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.382500 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.382777 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.383440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.385376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.396219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebaab0d-31bd-46fb-9f79-a24dbdf50b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.397074 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebaab0d-31bd-46fb-9f79-a24dbdf50b6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.397524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebaab0d-31bd-46fb-9f79-a24dbdf50b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.397888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebaab0d-31bd-46fb-9f79-a24dbdf50b6c admin admin] PATH_INFO: `/v3/projects/7f1368dc25aa4295b2f1439459495280/users/d24a310678c84aa3afca13113a964db1/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.407203 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1300/2578] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/users => generated 295 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:08.411223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.413725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebaab0d-31bd-46fb-9f79-a24dbdf50b6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f1368dc25aa4295b2f1439459495280, user_id=d24a310678c84aa3afca13113a964db1, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.416406 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebaab0d-31bd-46fb-9f79-a24dbdf50b6c admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.421561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b8cad7-5eb9-474c-978a-957e23287b0c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.426497 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.427068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.427354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.427633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.428274 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.429885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1279/2579] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:08 2022] PUT /identity/v3/projects/7f1368dc25aa4295b2f1439459495280/users/d24a310678c84aa3afca13113a964db1/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:08.430019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.440642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.440932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.441209 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.444856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1301/2580] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:08.448820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.458336 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebf093-487a-4b3d-81e4-e0afa838e8e3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.469276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] MFA Rules not processed for user `d24a310678c84aa3afca13113a964db1`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:08.470521 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:08.516612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.517272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.517551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.517825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] PATH_INFO: `/v3/projects/ab7c13c195634e87b3ea967850cd395c/users/b3b2e39327bf468cae9018574ad0873f/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.531968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab7c13c195634e87b3ea967850cd395c, user_id=b3b2e39327bf468cae9018574ad0873f, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.534612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.536587 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1280/2581] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:08.541741 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.549294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1302/2582] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:08 2022] PUT /identity/v3/projects/ab7c13c195634e87b3ea967850cd395c/users/b3b2e39327bf468cae9018574ad0873f/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:08.552923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee244ce-9847-4316-a44e-0e98e63cb917 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.553191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.612051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.612667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.612982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.613254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.614032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.616067 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.618229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806cbbc-261f-4be7-bbc1-857e4ce77852 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.622865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.623492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.623772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.624045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.624679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.626237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.635990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:08.636261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1303/2583] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:08.640324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.643972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1281/2584] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/projects => generated 389 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:08.647823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.651275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8626f30e-3ee1-4076-998b-fef365d7bf13 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.687855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:08.718884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d82c6-b7a9-43c2-b571-561533643aaf admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.719561 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20c1d92-9d39-4d3d-a264-2278f4257c73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.720230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c1d92-9d39-4d3d-a264-2278f4257c73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.720511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c1d92-9d39-4d3d-a264-2278f4257c73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.720783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c1d92-9d39-4d3d-a264-2278f4257c73 admin admin] PATH_INFO: `/v3/projects/ab7c13c195634e87b3ea967850cd395c/users/b3b2e39327bf468cae9018574ad0873f/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.722662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20c1d92-9d39-4d3d-a264-2278f4257c73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab7c13c195634e87b3ea967850cd395c, user_id=b3b2e39327bf468cae9018574ad0873f, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.723518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.724121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.724399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.724670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.725294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.725415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20c1d92-9d39-4d3d-a264-2278f4257c73 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.727195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.736201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1304/2585] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:08 2022] PUT /identity/v3/projects/ab7c13c195634e87b3ea967850cd395c/users/b3b2e39327bf468cae9018574ad0873f/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:08.746396 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.746765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.747385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.755464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1282/2586] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/users => generated 301 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:08.757549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.767928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdc9175-535e-47c3-ad1a-d8a8fe6b7c28 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.774876 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] MFA Rules not processed for user `b3b2e39327bf468cae9018574ad0873f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:08.776107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:08.826443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.827126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.827413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.827698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.828345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.829978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.839977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1283/2587] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:08.844229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.845197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1305/2588] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:08 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:08.854312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363033b7-26ea-4a22-9006-7cb4c9332044 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.857115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.858489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:08.859530 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.860205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.860487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.860758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] PATH_INFO: `/v3/projects/0519cba361574b659fdb15f00889cb3f/users/824e5397856c4044adee24516d44b450/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.876670 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0519cba361574b659fdb15f00889cb3f, user_id=824e5397856c4044adee24516d44b450, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.880077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.893594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1284/2589] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:08 2022] PUT /identity/v3/projects/0519cba361574b659fdb15f00889cb3f/users/824e5397856c4044adee24516d44b450/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:08.897719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.920009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3884f9-eaf1-44dc-9562-43a96a1f2098 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.962045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6234480-1c13-4df6-abd2-d3b0878aabc6 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.967669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.968280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.968609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:08.968732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.968961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.969357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:08.969602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:08.969705 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:08.969905 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:08.971214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:08.972473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:08.982318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1285/2590] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:08.986258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:08.996103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0210f039-123e-4831-aceb-97c945c02a37 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:08.999488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:09.000934 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97869b9a-b3b8-42bb-ad55-3559fc4d589e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:09.001585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97869b9a-b3b8-42bb-ad55-3559fc4d589e admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.001866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97869b9a-b3b8-42bb-ad55-3559fc4d589e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.002140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97869b9a-b3b8-42bb-ad55-3559fc4d589e admin admin] PATH_INFO: `/v3/projects/0519cba361574b659fdb15f00889cb3f/users/824e5397856c4044adee24516d44b450/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.004196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97869b9a-b3b8-42bb-ad55-3559fc4d589e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0519cba361574b659fdb15f00889cb3f, user_id=824e5397856c4044adee24516d44b450, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:09.006706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97869b9a-b3b8-42bb-ad55-3559fc4d589e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:09.018069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1286/2591] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:08 2022] PUT /identity/v3/projects/0519cba361574b659fdb15f00889cb3f/users/824e5397856c4044adee24516d44b450/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:09.029905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.030188 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.030458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.037552 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:09.039117 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:09.049131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1306/2592] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:08 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:09.057613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] MFA Rules not processed for user `824e5397856c4044adee24516d44b450`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:09.058813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:09.095351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:09.096880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:09.107631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8190cd4c-5ef1-43c2-b2f7-c9cae7c6ca3a service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:09.134735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1287/2593] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:09 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:09.146281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:09.207080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d0688c-cc57-437f-81ff-e3970a2c62c6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:09.210751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:09.211410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.211690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.211963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.222660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:09.224072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:09.239614 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1288/2594] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:09 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:09.285650 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:09.286351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.286521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.286825 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.288575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:09.349600 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:09.351226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683516a4-c433-4393-85b6-4c459794a147 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:09.360590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1307/2595] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:09 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:09.374523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:09.443938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c041b-2d99-451e-bd53-ccfa4f76f4a8 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:09.449258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:09.449892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.450791 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.451119 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] PATH_INFO: `/v3/users/d24a310678c84aa3afca13113a964db1` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.452378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d24a310678c84aa3afca13113a964db1)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:09.454240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:09.466941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:09.502139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd24a310678c84aa3afca13113a964db1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:09.509712 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd24a310678c84aa3afca13113a964db1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:09.522747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd24a310678c84aa3afca13113a964db1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:09.523427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd24a310678c84aa3afca13113a964db1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:09.530400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd24a310678c84aa3afca13113a964db1'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:09.536339 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1289/2596] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:09 2022] DELETE /identity/v3/users/d24a310678c84aa3afca13113a964db1 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:09.541136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.541630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.541904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.562904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:09.564146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:09.628018 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1308/2597] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:09.639197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:09.640682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:09.725703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea17980-342a-47f9-8c07-98620e26082b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:09.729463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:09.730100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:09.730378 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:09.730649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:09.740628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:09.742252 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:09.754645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1290/2598] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:10.022430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:10.023881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:10.040725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f36130-740d-4405-96dc-caa22c5706f4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:10.046437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:10.047487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.047816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.048267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] PATH_INFO: `/v3/projects/7f1368dc25aa4295b2f1439459495280` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.050399 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f1368dc25aa4295b2f1439459495280)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:10.053119 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:10.064470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:10.100487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f1368dc25aa4295b2f1439459495280', 'request_id': 'req-ff920611-d41b-45f5-9a43-b6885b19063b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.108015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f1368dc25aa4295b2f1439459495280', 'request_id': 'req-ff920611-d41b-45f5-9a43-b6885b19063b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.113487 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f1368dc25aa4295b2f1439459495280', 'request_id': 'req-ff920611-d41b-45f5-9a43-b6885b19063b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.124608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] The token cache is being invalidate because project 7f1368dc25aa4295b2f1439459495280 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:10.124894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.125515 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.127028 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1309/2599] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:10 2022] DELETE /identity/v3/projects/7f1368dc25aa4295b2f1439459495280 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:10.131469 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:10.141295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1124ea5-73ed-4274-b22f-5680dd99f4bf service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:10.197916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:10.198541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.198820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.199125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] PATH_INFO: `/v3/users/824e5397856c4044adee24516d44b450` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.200225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] RBAC: Authorizing `identity:delete_user(user_id=824e5397856c4044adee24516d44b450)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:10.202140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:10.243073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:10.254047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '824e5397856c4044adee24516d44b450'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.261597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '824e5397856c4044adee24516d44b450'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.273909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '824e5397856c4044adee24516d44b450'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.274506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '824e5397856c4044adee24516d44b450'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.282016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2f6afe9c-102c-4efd-b809-369b41ffd93a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '824e5397856c4044adee24516d44b450'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.288962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1291/2600] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:10 2022] DELETE /identity/v3/users/824e5397856c4044adee24516d44b450 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:10.551637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:10.553272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:10.566963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff920611-d41b-45f5-9a43-b6885b19063b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:10.623315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:10.624052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.624333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.624603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] PATH_INFO: `/v3/projects/0519cba361574b659fdb15f00889cb3f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.625694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0519cba361574b659fdb15f00889cb3f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:10.627663 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:10.672733 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0519cba361574b659fdb15f00889cb3f', 'request_id': 'req-7a0820d2-16ed-44ef-a4dd-2497bee3922d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.679922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0519cba361574b659fdb15f00889cb3f', 'request_id': 'req-7a0820d2-16ed-44ef-a4dd-2497bee3922d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.685352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0519cba361574b659fdb15f00889cb3f', 'request_id': 'req-7a0820d2-16ed-44ef-a4dd-2497bee3922d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.696261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] The token cache is being invalidate because project 0519cba361574b659fdb15f00889cb3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:10.696551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.697180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7a0820d2-16ed-44ef-a4dd-2497bee3922d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:10.698641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1310/2601] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:10 2022] DELETE /identity/v3/projects/0519cba361574b659fdb15f00889cb3f => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:10.704066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.704355 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.704628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.724282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:10.725498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:10.785352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1292/2602] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:10.793165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.793456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.793732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.815093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:10.816560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:10.824751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1311/2603] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:10.829344 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:10.839632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605442b2-c6f7-47ee-91f9-531714237b2b None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:10.844967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:10.845928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.846241 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.846513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.847212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:10.847558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:10.849153 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:10.864797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1293/2604] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:10.865989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:10.876204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941ec78e-9f90-4c9b-8fd5-c45854965a99 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:10.881355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:10.882049 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.882386 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.882719 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.883460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:10.885389 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:10.907845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:10.915583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1312/2605] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:10 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:10.919847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:10.929607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8ca90d-7417-4b8d-b78c-1f6db9d27523 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:10.992581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:10.993275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:10.993763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:10.994108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:10.994818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:10.996819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:11.020415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1294/2606] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:10 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:11.024786 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:11.035250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae147a1-7c2c-4678-86f2-604ed7bb7f6f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:11.040039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:11.040667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:11.040946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:11.041222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:11.041863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:11.043479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:11.058972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1313/2607] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:11.061179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:11.073916 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59cff33-fafe-4ee7-92aa-4d7a6cf85df8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:11.081009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6283c2-14e5-4a57-993f-b49c9cdb4734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:11.081864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6283c2-14e5-4a57-993f-b49c9cdb4734 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:11.081989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6283c2-14e5-4a57-993f-b49c9cdb4734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:11.082263 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6283c2-14e5-4a57-993f-b49c9cdb4734 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:11.083043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6283c2-14e5-4a57-993f-b49c9cdb4734 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:11.084551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6283c2-14e5-4a57-993f-b49c9cdb4734 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:11.095852 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1295/2608] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:11.100054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:11.110530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce41bcf3-eb54-4468-abfb-b932daafbe54 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:11.115607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:11.116320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:11.116634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:11.116875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] PATH_INFO: `/v3/projects/ee36270279f94439a30b92a01c139495/users/94ef41f9745e46789547c0a7fbe35bed/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:11.133460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee36270279f94439a30b92a01c139495, user_id=94ef41f9745e46789547c0a7fbe35bed, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:11.136192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:11.150793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1314/2609] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:11 2022] PUT /identity/v3/projects/ee36270279f94439a30b92a01c139495/users/94ef41f9745e46789547c0a7fbe35bed/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:11.161432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:11.161978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:11.161978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:11.192376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] MFA Rules not processed for user `94ef41f9745e46789547c0a7fbe35bed`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:11.193774 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:11.199366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:11.261083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1296/2610] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:11.272235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:11.273939 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:11.332011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65268997-4c96-4287-8ed7-c93d903c57ff admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:11.335653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:11.336250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:11.336524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:11.336792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:11.345060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:11.346455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:11.354994 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1315/2611] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:12.062140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:12.063985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:12.138879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cca0027-8fb6-466c-a564-60c3c4299629 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:12.144335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:12.144942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:12.145236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:12.145490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:12.147269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:12.213096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:12.214709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c523247e-de5e-42f5-a6d6-a744be0dc037 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:12.225475 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1297/2612] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:12 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:12.285227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:12.287030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:12.358138 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2435fd-2c3f-443f-adee-2e661139bebf service glance] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:12.366481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:12.367176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:12.367473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:12.367865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] PATH_INFO: `/v3/users/4c99ad129b1a4ec19867e54370ec8c30` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:12.368886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c99ad129b1a4ec19867e54370ec8c30)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:12.370783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:12.419664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c99ad129b1a4ec19867e54370ec8c30'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:12.427548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c99ad129b1a4ec19867e54370ec8c30'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:12.438741 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:12.440280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c99ad129b1a4ec19867e54370ec8c30'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:12.441129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c99ad129b1a4ec19867e54370ec8c30'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:12.450857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c99ad129b1a4ec19867e54370ec8c30'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:12.456941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1316/2613] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:12 2022] DELETE /identity/v3/users/4c99ad129b1a4ec19867e54370ec8c30 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:12.462910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:12.463483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:12.463758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:12.502843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:12.504396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:12.555505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:12.578158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1298/2614] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:12.588345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:12.589804 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:12.651748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c051d7d-8f3f-4fd0-84a9-22664be05aa6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:12.655947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:12.656553 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:12.656830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:12.657104 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:12.665997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:12.667448 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:12.677509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1317/2615] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:12.967903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:12.969471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:12.981810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdbb428-2c6a-4a23-b9de-6c374fbabb66 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:12.986602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:12.987276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:12.987554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:12.987833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] PATH_INFO: `/v3/projects/21c35f3c9b6149abaa8d7cb44566db18` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:12.988977 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] RBAC: Authorizing `identity:delete_project(project_id=21c35f3c9b6149abaa8d7cb44566db18)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:12.990911 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:13.039497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21c35f3c9b6149abaa8d7cb44566db18', 'request_id': 'req-abb75999-6463-4db6-b8c8-25bc54ddfe9b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.047127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21c35f3c9b6149abaa8d7cb44566db18', 'request_id': 'req-abb75999-6463-4db6-b8c8-25bc54ddfe9b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.052947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21c35f3c9b6149abaa8d7cb44566db18', 'request_id': 'req-abb75999-6463-4db6-b8c8-25bc54ddfe9b'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.066781 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] The token cache is being invalidate because project 21c35f3c9b6149abaa8d7cb44566db18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:13.067141 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.067945 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.069932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1299/2616] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:12 2022] DELETE /identity/v3/projects/21c35f3c9b6149abaa8d7cb44566db18 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:13.074252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:13.084753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09326231-f38b-457b-b958-e4eaebbb88e6 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:13.140559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:13.141177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:13.141454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:13.141726 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] PATH_INFO: `/v3/users/f7743fd355ae43c9ae6d68cd799faeca` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:13.142834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7743fd355ae43c9ae6d68cd799faeca)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:13.144797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:13.191999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7743fd355ae43c9ae6d68cd799faeca'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.200647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7743fd355ae43c9ae6d68cd799faeca'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.213964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7743fd355ae43c9ae6d68cd799faeca'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.214835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7743fd355ae43c9ae6d68cd799faeca'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.222160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7743fd355ae43c9ae6d68cd799faeca'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.228354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1318/2617] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:13 2022] DELETE /identity/v3/users/f7743fd355ae43c9ae6d68cd799faeca => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:13.526967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:13.527500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:13.549131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb75999-6463-4db6-b8c8-25bc54ddfe9b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:13.615132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:13.615132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:13.615132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:13.615132 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] PATH_INFO: `/v3/projects/adb69ccfbc7f4dc2bcf109bc535860b8` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:13.615478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] RBAC: Authorizing `identity:delete_project(project_id=adb69ccfbc7f4dc2bcf109bc535860b8)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:13.616361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:13.669721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adb69ccfbc7f4dc2bcf109bc535860b8', 'request_id': 'req-45a08d73-1821-4e76-b94c-bba67ac0e419'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.679803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adb69ccfbc7f4dc2bcf109bc535860b8', 'request_id': 'req-45a08d73-1821-4e76-b94c-bba67ac0e419'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.688091 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adb69ccfbc7f4dc2bcf109bc535860b8', 'request_id': 'req-45a08d73-1821-4e76-b94c-bba67ac0e419'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.701782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:13.709829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] The token cache is being invalidate because project adb69ccfbc7f4dc2bcf109bc535860b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:13.710540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.711294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.713113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1300/2618] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:13 2022] DELETE /identity/v3/projects/adb69ccfbc7f4dc2bcf109bc535860b8 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:13.717010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:13.718447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:13.731778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578eb487-7136-4894-87ee-28d41a5d5cd8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:13.737222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:13.797011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:13.797634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:13.797869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:13.798136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] PATH_INFO: `/v3/users/b3b2e39327bf468cae9018574ad0873f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:13.799412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3b2e39327bf468cae9018574ad0873f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:13.801319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:13.850746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3b2e39327bf468cae9018574ad0873f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.860879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3b2e39327bf468cae9018574ad0873f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.874051 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3b2e39327bf468cae9018574ad0873f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.874755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3b2e39327bf468cae9018574ad0873f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.882578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-59dfce16-a1da-4920-bced-b03896ad9bc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3b2e39327bf468cae9018574ad0873f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:13.890446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1319/2619] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:13 2022] DELETE /identity/v3/users/b3b2e39327bf468cae9018574ad0873f => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:14.193886 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.196148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:14.210594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08d73-1821-4e76-b94c-bba67ac0e419 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.271246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.271890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.272173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.272451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] PATH_INFO: `/v3/projects/ab7c13c195634e87b3ea967850cd395c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.273576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab7c13c195634e87b3ea967850cd395c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.275562 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.321440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab7c13c195634e87b3ea967850cd395c', 'request_id': 'req-4e506ebd-2d30-4710-ac59-e631298b466d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:14.328780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab7c13c195634e87b3ea967850cd395c', 'request_id': 'req-4e506ebd-2d30-4710-ac59-e631298b466d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:14.334356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab7c13c195634e87b3ea967850cd395c', 'request_id': 'req-4e506ebd-2d30-4710-ac59-e631298b466d'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:14.345867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] The token cache is being invalidate because project ab7c13c195634e87b3ea967850cd395c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:14.346165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:14.346800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4e506ebd-2d30-4710-ac59-e631298b466d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:14.348328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1301/2620] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:14 2022] DELETE /identity/v3/projects/ab7c13c195634e87b3ea967850cd395c => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:14.354168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.354454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.354729 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.378352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:14.379603 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:14.442798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1320/2621] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:14.452612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.452895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.453167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.473342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:14.474533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:14.482487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1302/2622] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:14.486828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.496431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0d4065-df6c-4db8-97c0-2f6729a1a510 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.500829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.501418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.501694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.501966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.502601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:14.502900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.504526 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.516265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1321/2623] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:14.520612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.530483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8e023c-9e8b-4d69-aa9b-dd6eeff80bf6 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.535320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.535907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.536185 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.536452 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.537082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.538996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.565607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1303/2624] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:14 2022] POST /identity/v3/projects => generated 367 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:14.570419 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.580848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9b5271-6177-4404-8246-b2237686dcd4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.593492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:14.642794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.643435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.643713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.643983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.644605 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.646493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.669183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1322/2625] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:14 2022] POST /identity/v3/users => generated 284 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:14.674058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.685612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2e623e-cfea-4ea2-a87b-bcb22b7991f1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.690352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.690980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.691536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.691821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.692464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.694045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.704515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1304/2626] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:14.709215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.720800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9fc2be-df40-4e3f-8410-ae0bbb017215 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.726084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054378fe-4243-4135-a0da-daa9d114041d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.726792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054378fe-4243-4135-a0da-daa9d114041d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.727126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054378fe-4243-4135-a0da-daa9d114041d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.727404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054378fe-4243-4135-a0da-daa9d114041d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.728064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054378fe-4243-4135-a0da-daa9d114041d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.729682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054378fe-4243-4135-a0da-daa9d114041d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.741673 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1323/2627] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:14.745786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.756307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d390c4e5-9330-4b62-934a-7a76e5598fbf admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.760849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.761484 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.761759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.762030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] PATH_INFO: `/v3/projects/e3895590ccff458b9b1eb8d30c0fae01/users/359377cac1e844c8898efa88e5cadd7d/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.776696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3895590ccff458b9b1eb8d30c0fae01, user_id=359377cac1e844c8898efa88e5cadd7d, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.779379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:14.792257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1305/2628] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:14 2022] PUT /identity/v3/projects/e3895590ccff458b9b1eb8d30c0fae01/users/359377cac1e844c8898efa88e5cadd7d/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:14.803275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.803581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.803865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.833562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] MFA Rules not processed for user `359377cac1e844c8898efa88e5cadd7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:14.834640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:14.899640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1324/2629] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:14.904312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:14.932325 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:14.969613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f544fd-4da2-42cb-888d-c40a13547bd0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:14.974353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:14.974990 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:14.975279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:14.975555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:14.976175 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:14.978058 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.008167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1306/2630] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:14 2022] POST /identity/v3/projects => generated 365 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:15.012150 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.022892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee7d31a-8fc6-4263-8772-8f97ff497ed4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.080068 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.080668 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.080945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.081212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.081840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.083823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.105830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1325/2631] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:15 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:15.110216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.120267 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38466228-69d6-46bf-8780-da049c9efcb6 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.125399 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.125994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.126272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.126541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.127211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.128792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.139068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1307/2632] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:15.143101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.153558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa2a3ec-5af2-4003-9a72-86f3132f7dd3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.158395 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.159099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.159379 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.159654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] PATH_INFO: `/v3/projects/bfe70136c7024f0e8cdb24175c0be615/users/05820677f59d47f98222e3f6557c46e8/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.172124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:15.176881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe70136c7024f0e8cdb24175c0be615, user_id=05820677f59d47f98222e3f6557c46e8, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.180010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.193609 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1326/2633] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:15 2022] PUT /identity/v3/projects/bfe70136c7024f0e8cdb24175c0be615/users/05820677f59d47f98222e3f6557c46e8/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:15.197985 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.207959 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebbf1ae-4569-4aca-9a56-cbb07c1d292e admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.264545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d530b-a8fa-483b-b15b-6f71fc546197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.265151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d530b-a8fa-483b-b15b-6f71fc546197 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.265430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d530b-a8fa-483b-b15b-6f71fc546197 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.265701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d530b-a8fa-483b-b15b-6f71fc546197 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.266341 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d530b-a8fa-483b-b15b-6f71fc546197 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.267993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d530b-a8fa-483b-b15b-6f71fc546197 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.277854 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1308/2634] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:15 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:15.282006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.292005 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e935c7-47ad-4ae9-85d9-588644db4f7f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.297196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.298006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.298140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.298401 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] PATH_INFO: `/v3/projects/bfe70136c7024f0e8cdb24175c0be615/users/05820677f59d47f98222e3f6557c46e8/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.300580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe70136c7024f0e8cdb24175c0be615, user_id=05820677f59d47f98222e3f6557c46e8, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.303242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.315120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1327/2635] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:15 2022] PUT /identity/v3/projects/bfe70136c7024f0e8cdb24175c0be615/users/05820677f59d47f98222e3f6557c46e8/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:15.325506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1310189d-8db1-4543-b864-1222696265f2 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.325796 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1310189d-8db1-4543-b864-1222696265f2 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.326072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1310189d-8db1-4543-b864-1222696265f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.354195 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-1310189d-8db1-4543-b864-1222696265f2 None None] MFA Rules not processed for user `05820677f59d47f98222e3f6557c46e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:15.355367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1310189d-8db1-4543-b864-1222696265f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:15.422337 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1309/2636] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:15 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:15.434462 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.436240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:15.492795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54994dd-4e0e-4bfb-80ae-ec598a732174 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.496627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.497233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.497513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.497783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.507097 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.508596 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.518048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1328/2637] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:15 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:15.599969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1310189d-8db1-4543-b864-1222696265f2 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:15.611497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1310189d-8db1-4543-b864-1222696265f2 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:15.615279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5037a1f6-2e5c-41b0-a623-52494f9566ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:15.615823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5037a1f6-2e5c-41b0-a623-52494f9566ec service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:15.616118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5037a1f6-2e5c-41b0-a623-52494f9566ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:15.617740 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5037a1f6-2e5c-41b0-a623-52494f9566ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:15.636759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:15.681228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5037a1f6-2e5c-41b0-a623-52494f9566ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:15.683065 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5037a1f6-2e5c-41b0-a623-52494f9566ec service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:15.694061 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1310/2638] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:15 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:20.447345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:20.510751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356d536a-b9fb-4c1d-8f42-0e9b1f99aa32 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:20.515562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:20.516191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:20.516468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:20.516743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] PATH_INFO: `/v3/users/359377cac1e844c8898efa88e5cadd7d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:20.517897 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] RBAC: Authorizing `identity:delete_user(user_id=359377cac1e844c8898efa88e5cadd7d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:20.519870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:20.568485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '359377cac1e844c8898efa88e5cadd7d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:20.580512 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '359377cac1e844c8898efa88e5cadd7d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:20.593421 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '359377cac1e844c8898efa88e5cadd7d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:20.594069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '359377cac1e844c8898efa88e5cadd7d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:20.601286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '359377cac1e844c8898efa88e5cadd7d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:20.602811 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:20.609195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1329/2639] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:20 2022] DELETE /identity/v3/users/359377cac1e844c8898efa88e5cadd7d => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:20.613837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:20.614402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:20.614517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:20.634657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:20.635996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:20.697574 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1311/2640] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:20.709078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:20.710619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:20.767632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bfd3a5-c8e1-48c3-8d5a-765aadeef801 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:20.771190 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:20.771793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:20.772073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:20.772349 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:20.781229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:20.782638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:20.792258 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1330/2641] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:21.063664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:21.065116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:21.077035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe50b0a-9c51-4a6c-9b37-2dac1a67b077 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:21.081599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:21.082229 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:21.082505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:21.082777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] PATH_INFO: `/v3/projects/e3895590ccff458b9b1eb8d30c0fae01` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:21.083926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3895590ccff458b9b1eb8d30c0fae01)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:21.085825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:21.132793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3895590ccff458b9b1eb8d30c0fae01', 'request_id': 'req-5f6c74dd-7059-47c8-a0e1-8226b354e460'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.140690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3895590ccff458b9b1eb8d30c0fae01', 'request_id': 'req-5f6c74dd-7059-47c8-a0e1-8226b354e460'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.151101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3895590ccff458b9b1eb8d30c0fae01', 'request_id': 'req-5f6c74dd-7059-47c8-a0e1-8226b354e460'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.163719 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] The token cache is being invalidate because project e3895590ccff458b9b1eb8d30c0fae01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:21.164012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.164671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.166157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1312/2642] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:21 2022] DELETE /identity/v3/projects/e3895590ccff458b9b1eb8d30c0fae01 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:21.170772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:21.180382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b77036-cabe-4836-a780-a81c669548f0 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:21.240378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:21.241007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:21.241290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:21.241564 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] PATH_INFO: `/v3/users/05820677f59d47f98222e3f6557c46e8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:21.242682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05820677f59d47f98222e3f6557c46e8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:21.244637 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:21.298480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05820677f59d47f98222e3f6557c46e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.307280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05820677f59d47f98222e3f6557c46e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.320449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05820677f59d47f98222e3f6557c46e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.321133 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05820677f59d47f98222e3f6557c46e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.328383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c5bfb9e8-aa9b-42e2-84da-081c846ed6e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05820677f59d47f98222e3f6557c46e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.334105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1331/2643] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:21 2022] DELETE /identity/v3/users/05820677f59d47f98222e3f6557c46e8 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:21.649715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:21.651379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:21.667828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6c74dd-7059-47c8-a0e1-8226b354e460 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:21.719179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:21.737350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:21.737982 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:21.738262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:21.738531 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] PATH_INFO: `/v3/projects/bfe70136c7024f0e8cdb24175c0be615` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:21.739777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfe70136c7024f0e8cdb24175c0be615)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:21.746246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:21.795507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfe70136c7024f0e8cdb24175c0be615', 'request_id': 'req-342757cf-5ff5-4537-b06a-138530e140b7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.804278 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfe70136c7024f0e8cdb24175c0be615', 'request_id': 'req-342757cf-5ff5-4537-b06a-138530e140b7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.810012 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfe70136c7024f0e8cdb24175c0be615', 'request_id': 'req-342757cf-5ff5-4537-b06a-138530e140b7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.823523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] The token cache is being invalidate because project bfe70136c7024f0e8cdb24175c0be615 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:21.823779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.824446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-342757cf-5ff5-4537-b06a-138530e140b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:21.826596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1313/2644] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:21 2022] DELETE /identity/v3/projects/bfe70136c7024f0e8cdb24175c0be615 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:21.831468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:21.831944 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:21.832158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:21.857934 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:21.859201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:21.873201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:21.928433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1332/2645] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:21.936708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:21.936991 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:21.937292 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:21.958568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:21.959813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:21.967934 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1314/2646] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:21.972607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:21.982483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071e50e8-255b-4f3f-8533-f26b33ae6051 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:21.987155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:21.987753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:21.988032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:21.988305 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:21.988948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:21.989249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:21.990821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.003209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1333/2647] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:22.008371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.018820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6375e4-81f6-432e-961a-050f0749b96b None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.023493 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.024098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.024374 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.024642 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.025304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.027217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.053728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1315/2648] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:22 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:22.057984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.068627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f3e4d8-9913-4c46-86da-788e03c4047f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.124085 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.124731 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.125008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.125276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.125902 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.127868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.151161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1334/2649] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:22 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:22.156135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.166231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35047c8-f824-4371-afce-bdb890f44496 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.170834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.171458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.171737 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.172006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.172645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.174219 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.184732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1316/2650] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:22.188627 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.198807 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60140cf-b2ed-4499-be09-625520aeb9fb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.204215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e739c3-86e2-491b-b5c7-f2186b503cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.204917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e739c3-86e2-491b-b5c7-f2186b503cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.205248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e739c3-86e2-491b-b5c7-f2186b503cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.205610 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e739c3-86e2-491b-b5c7-f2186b503cc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.206260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e739c3-86e2-491b-b5c7-f2186b503cc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.207960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e739c3-86e2-491b-b5c7-f2186b503cc3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.219360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1335/2651] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:22.223480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.233333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2685ac-fbe8-4fc9-86f5-a22dd34fec42 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.237905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.238581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.238856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.239183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] PATH_INFO: `/v3/projects/09a3f13109ca4ad791ff77629b380bec/users/b2bf913a80c04cebab9f68e6b26cd7ca/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.254025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09a3f13109ca4ad791ff77629b380bec, user_id=b2bf913a80c04cebab9f68e6b26cd7ca, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.256700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.272214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1317/2652] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:22 2022] PUT /identity/v3/projects/09a3f13109ca4ad791ff77629b380bec/users/b2bf913a80c04cebab9f68e6b26cd7ca/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:22.283524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.284101 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.284681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.316231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] MFA Rules not processed for user `b2bf913a80c04cebab9f68e6b26cd7ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:22.317980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:22.406464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1336/2653] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:22.410695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.477723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c216a27-8be3-4a24-b20b-31e432f2e486 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.482398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.483031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.483313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.483588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.484213 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.486104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.501842 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:22.520596 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1318/2654] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:22 2022] POST /identity/v3/projects => generated 373 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:22.521772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.533388 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e04500-a9a1-486a-ba61-08db59876279 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.545366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:22.594740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.595486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.595762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.596032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.596657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.598583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.619629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1337/2655] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:22 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:22.623695 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.633910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c277058b-480c-4542-a3d2-3dd5b606ccb1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.638777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.639554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.639837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.640111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.640751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.642350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.656550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1319/2656] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:22.657692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.670476 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bb552c-2ddd-44da-9b44-45ecdda38279 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.675707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.676702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.676981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.677294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] PATH_INFO: `/v3/projects/ed5f783412d644edb818ad7c40f747a7/users/7e5acbc863544075bf3d6a9a4bbc1708/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.693200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed5f783412d644edb818ad7c40f747a7, user_id=7e5acbc863544075bf3d6a9a4bbc1708, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.695848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.708485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1338/2657] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:22 2022] PUT /identity/v3/projects/ed5f783412d644edb818ad7c40f747a7/users/7e5acbc863544075bf3d6a9a4bbc1708/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:22.712430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.723244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2a039-ef6f-46ab-a808-a48d82ec7074 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.783650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027c2c4b-b5f6-47cf-8731-628bd4d97232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.784270 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c2c4b-b5f6-47cf-8731-628bd4d97232 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.784554 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c2c4b-b5f6-47cf-8731-628bd4d97232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.784832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c2c4b-b5f6-47cf-8731-628bd4d97232 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.785483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027c2c4b-b5f6-47cf-8731-628bd4d97232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.787129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027c2c4b-b5f6-47cf-8731-628bd4d97232 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.797372 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1320/2658] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:22 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:22.801681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:22.812393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470dae65-e3f7-443f-a2c3-5a679a838476 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:22.817322 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:22.817969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.818251 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.818522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] PATH_INFO: `/v3/projects/ed5f783412d644edb818ad7c40f747a7/users/7e5acbc863544075bf3d6a9a4bbc1708/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.820456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed5f783412d644edb818ad7c40f747a7, user_id=7e5acbc863544075bf3d6a9a4bbc1708, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:22.823076 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:22.834327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1339/2659] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:22 2022] PUT /identity/v3/projects/ed5f783412d644edb818ad7c40f747a7/users/7e5acbc863544075bf3d6a9a4bbc1708/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:22.845294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:22.845586 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:22.845861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:22.874794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] MFA Rules not processed for user `7e5acbc863544075bf3d6a9a4bbc1708`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:22.876027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:23.066004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1321/2660] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:22 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:23.077196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:23.078841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:23.143348 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a81e96c-9433-4558-899b-218f02e68a06 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:23.147756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5c09d6-7347-4d3f-a8b3-0267a906c375 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:23.148485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5c09d6-7347-4d3f-a8b3-0267a906c375 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:23.148844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5c09d6-7347-4d3f-a8b3-0267a906c375 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:23.149191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5c09d6-7347-4d3f-a8b3-0267a906c375 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:23.211290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5c09d6-7347-4d3f-a8b3-0267a906c375 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:23.212728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5c09d6-7347-4d3f-a8b3-0267a906c375 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:23.222383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1340/2661] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:25.762730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:25.827097 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd42b62b-7ec7-4cc2-b9fc-1577fba140c3 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:25.831795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:25.832431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:25.832713 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:25.832989 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] PATH_INFO: `/v3/users/b2bf913a80c04cebab9f68e6b26cd7ca` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:25.834143 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2bf913a80c04cebab9f68e6b26cd7ca)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:25.836233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:25.841100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:25.884915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2bf913a80c04cebab9f68e6b26cd7ca'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:25.892593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2bf913a80c04cebab9f68e6b26cd7ca'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:25.905227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2bf913a80c04cebab9f68e6b26cd7ca'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:25.906490 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2bf913a80c04cebab9f68e6b26cd7ca'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:25.913780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2bf913a80c04cebab9f68e6b26cd7ca'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:25.920137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1322/2662] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:25 2022] DELETE /identity/v3/users/b2bf913a80c04cebab9f68e6b26cd7ca => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:25.925309 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7081c786-6571-4813-9038-50af435bfb28 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:25.925830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7081c786-6571-4813-9038-50af435bfb28 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:25.925929 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7081c786-6571-4813-9038-50af435bfb28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:25.944849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7081c786-6571-4813-9038-50af435bfb28 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:25.946048 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7081c786-6571-4813-9038-50af435bfb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:25.963015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:26.011365 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1341/2663] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:26.022434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:26.023884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:26.113400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e0cf97-d3cb-4836-a79b-ad3a6fe70b57 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:26.117416 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:26.118076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:26.118356 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:26.118777 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:26.130621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:26.132106 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:26.145222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1323/2664] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:26.442997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7081c786-6571-4813-9038-50af435bfb28 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:26.444652 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7081c786-6571-4813-9038-50af435bfb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:26.459664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7081c786-6571-4813-9038-50af435bfb28 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:26.464712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:26.465360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:26.465710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:26.465985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] PATH_INFO: `/v3/projects/09a3f13109ca4ad791ff77629b380bec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:26.467272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09a3f13109ca4ad791ff77629b380bec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:26.469879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:26.522044 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09a3f13109ca4ad791ff77629b380bec', 'request_id': 'req-f0230335-1b15-476f-a279-fa5b6ce75303'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.529759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09a3f13109ca4ad791ff77629b380bec', 'request_id': 'req-f0230335-1b15-476f-a279-fa5b6ce75303'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.536311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09a3f13109ca4ad791ff77629b380bec', 'request_id': 'req-f0230335-1b15-476f-a279-fa5b6ce75303'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.547687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] The token cache is being invalidate because project 09a3f13109ca4ad791ff77629b380bec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:26.547921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.548601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.552881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1342/2665] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:26 2022] DELETE /identity/v3/projects/09a3f13109ca4ad791ff77629b380bec => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:26.554601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:26.565651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf70768-40c3-4a73-a471-fcdc5825866a service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:26.627128 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:26.627833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:26.628178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:26.628511 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] PATH_INFO: `/v3/users/7e5acbc863544075bf3d6a9a4bbc1708` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:26.629728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e5acbc863544075bf3d6a9a4bbc1708)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:26.631754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:26.670723 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:26.695215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e5acbc863544075bf3d6a9a4bbc1708'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.713963 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e5acbc863544075bf3d6a9a4bbc1708'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.749328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e5acbc863544075bf3d6a9a4bbc1708'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.755421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e5acbc863544075bf3d6a9a4bbc1708'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.775316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-beec97a0-12cc-4c95-ac13-a09b2dc1bd14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e5acbc863544075bf3d6a9a4bbc1708'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:26.782682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1324/2666] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:26 2022] DELETE /identity/v3/users/7e5acbc863544075bf3d6a9a4bbc1708 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:27.064700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.066983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:27.082457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0230335-1b15-476f-a279-fa5b6ce75303 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.144985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.145615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.145888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.146154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] PATH_INFO: `/v3/projects/ed5f783412d644edb818ad7c40f747a7` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.147315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed5f783412d644edb818ad7c40f747a7)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.149213 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.195997 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:27.197473 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed5f783412d644edb818ad7c40f747a7', 'request_id': 'req-35bb868f-9a69-4d42-91b6-630390c74d4f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:27.205891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed5f783412d644edb818ad7c40f747a7', 'request_id': 'req-35bb868f-9a69-4d42-91b6-630390c74d4f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:27.212824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed5f783412d644edb818ad7c40f747a7', 'request_id': 'req-35bb868f-9a69-4d42-91b6-630390c74d4f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:27.225459 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] The token cache is being invalidate because project ed5f783412d644edb818ad7c40f747a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:27.225752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:27.226404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35bb868f-9a69-4d42-91b6-630390c74d4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:27.227915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1343/2667] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:27 2022] DELETE /identity/v3/projects/ed5f783412d644edb818ad7c40f747a7 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:27.232964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.233322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.233666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.255580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:27.258451 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:27.315873 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1325/2668] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:27.323848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.324134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.324408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.344297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:27.345494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:27.353291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1344/2669] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:27.357423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.367803 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d39c59-96f5-4ed3-8c95-c65d034aa9a5 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.372350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.372942 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.373222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.373497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.374139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:27.374444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.376082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.387400 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1326/2670] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:27.391430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.400503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b013587-aa64-42b3-bade-2a59adfffe96 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.405102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.405683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.406481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.406758 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.407491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.409397 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.437245 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1345/2671] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:27.441794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.451710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa12e1ea-b039-4d5d-89cf-66ed3fe9b9ae admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.507149 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.507745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.508030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.508307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.508932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.510837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.531564 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1327/2672] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/users => generated 289 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:27.536261 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.546168 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a301986-e103-4ace-a478-52ea65c3c6d1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.551376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.551961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.552235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.552511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.553144 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.554703 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.564408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1346/2673] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:27.568499 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.578103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cad757b-c901-46f8-9337-c593966899c0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.582759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539983b-6f9c-4285-ab1c-10bfacea5192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.583380 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539983b-6f9c-4285-ab1c-10bfacea5192 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.583666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539983b-6f9c-4285-ab1c-10bfacea5192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.583941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539983b-6f9c-4285-ab1c-10bfacea5192 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.584580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539983b-6f9c-4285-ab1c-10bfacea5192 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.586154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539983b-6f9c-4285-ab1c-10bfacea5192 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.596274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1328/2674] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:27.600090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.609803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3cd4b5-2852-462c-bbd3-3993b9e8f09f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.614890 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.615545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.615858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.616077 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] PATH_INFO: `/v3/projects/9f711a760ebb4f7486df1deb767373c7/users/59667bb9f8be401fab0d5a9fcd16ff7b/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.631453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f711a760ebb4f7486df1deb767373c7, user_id=59667bb9f8be401fab0d5a9fcd16ff7b, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.634063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.647962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1347/2675] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:27 2022] PUT /identity/v3/projects/9f711a760ebb4f7486df1deb767373c7/users/59667bb9f8be401fab0d5a9fcd16ff7b/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:27.658794 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.659108 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.659426 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.681127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:27.689449 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] MFA Rules not processed for user `59667bb9f8be401fab0d5a9fcd16ff7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:27.691331 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:27.763039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1329/2676] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:27.767988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.832232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1247430-0652-4cc5-ba97-06e9c9637ce9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.839437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.839437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.839437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.839437 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.841122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.841122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.869551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1348/2677] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/projects => generated 375 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:27.873464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.884183 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce993c2c-5937-4036-b2c0-9b2c8993ad8d None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.941971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.942571 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.942848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.943151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.943780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.945673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:27.967797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1330/2678] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:27 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:27.972424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:27.982909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3435a728-d976-4e72-be78-5eed2b39b968 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:27.987573 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:27.988237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:27.988981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:27.988981 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:27.989538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:27.991240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.002357 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1349/2679] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:28.006674 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.017797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc9cf1-1c3e-4404-8a9c-3ad8f7bc6320 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.033228 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9084292-2c93-4ac1-8605-82ebeab2372d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.033922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9084292-2c93-4ac1-8605-82ebeab2372d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.034146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9084292-2c93-4ac1-8605-82ebeab2372d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.034420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9084292-2c93-4ac1-8605-82ebeab2372d admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.035137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9084292-2c93-4ac1-8605-82ebeab2372d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.036776 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9084292-2c93-4ac1-8605-82ebeab2372d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.047500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1331/2680] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:28.051666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.061716 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95aa2c-fd9f-458b-856a-abb479e36184 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.067105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.067746 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.068035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.068548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] PATH_INFO: `/v3/projects/df202069e3b1467c940f64d4b8b89904/users/4162d42730b64a65b7a96e3b5323c1d3/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.083711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df202069e3b1467c940f64d4b8b89904, user_id=4162d42730b64a65b7a96e3b5323c1d3, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.086290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.099263 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1350/2681] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:28 2022] PUT /identity/v3/projects/df202069e3b1467c940f64d4b8b89904/users/4162d42730b64a65b7a96e3b5323c1d3/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:28.169015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.169367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.169731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.199100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] MFA Rules not processed for user `4162d42730b64a65b7a96e3b5323c1d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:28.200321 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:28.272022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1332/2682] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:28.276775 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.303617 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:28.345422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d90fc2-f3f7-401b-b4fc-8152c43f7074 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.349984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.350591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.350878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.351186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.351835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.353760 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.382964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1351/2683] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:28 2022] POST /identity/v3/projects => generated 373 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:28.386233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.397823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b825db0-115a-4c08-9d99-5876779d0570 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.402968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:28.459506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.460120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.460401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.460669 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.461311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.463240 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.484900 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1333/2684] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:28 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:28.489369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.499785 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8dabd-780a-4d34-b0c1-fd4bdd4916af admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.505013 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.505618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.505896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.506173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.506904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.508622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.520159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1352/2685] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:28.524798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.534661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab65803-a5f1-410a-a6a5-77863f08f390 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.539314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.540064 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.540218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.540492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] PATH_INFO: `/v3/projects/953cfadb5c4a4d2b8df7dc4604b6cd6d/users/66daa5467c7b46a8a8b647244c9a3675/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.556178 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=953cfadb5c4a4d2b8df7dc4604b6cd6d, user_id=66daa5467c7b46a8a8b647244c9a3675, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.558901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.573272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1334/2686] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:28 2022] PUT /identity/v3/projects/953cfadb5c4a4d2b8df7dc4604b6cd6d/users/66daa5467c7b46a8a8b647244c9a3675/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:28.577278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.587822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a6e18-bd6b-4d88-ab2b-29953f32df45 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.647164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4049eaa-b8fc-44b1-a1e0-537e3caf0f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.647763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4049eaa-b8fc-44b1-a1e0-537e3caf0f02 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.648041 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4049eaa-b8fc-44b1-a1e0-537e3caf0f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.648352 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4049eaa-b8fc-44b1-a1e0-537e3caf0f02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.648989 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4049eaa-b8fc-44b1-a1e0-537e3caf0f02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.650588 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4049eaa-b8fc-44b1-a1e0-537e3caf0f02 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.660208 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1353/2687] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:28.665158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.676376 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771cc059-8fab-4395-b5c6-7f1244bb8cc8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.681105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.681754 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.682029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.682299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] PATH_INFO: `/v3/projects/953cfadb5c4a4d2b8df7dc4604b6cd6d/users/66daa5467c7b46a8a8b647244c9a3675/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.684217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=953cfadb5c4a4d2b8df7dc4604b6cd6d, user_id=66daa5467c7b46a8a8b647244c9a3675, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.686828 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.698231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1335/2688] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:28 2022] PUT /identity/v3/projects/953cfadb5c4a4d2b8df7dc4604b6cd6d/users/66daa5467c7b46a8a8b647244c9a3675/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:28.708436 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.708728 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.709006 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.739829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] MFA Rules not processed for user `66daa5467c7b46a8a8b647244c9a3675`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:28.741045 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:28.820278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1354/2689] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:28.831950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:28.833815 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:28.895680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c3adca-d6ac-4ebe-99de-767794531492 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:28.899964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0118428d-7f24-4401-8230-764bc93647d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:28.900573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0118428d-7f24-4401-8230-764bc93647d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:28.900859 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0118428d-7f24-4401-8230-764bc93647d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:28.901130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0118428d-7f24-4401-8230-764bc93647d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:28.960838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0118428d-7f24-4401-8230-764bc93647d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:28.962273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0118428d-7f24-4401-8230-764bc93647d5 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:28.975038 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1336/2690] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:30.109823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:30.111841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.132282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:30.178030 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee56cba-548c-4ea2-97c7-42ada66da57e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:30.182504 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:30.183159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:30.183438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:30.183708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] PATH_INFO: `/v3/users/94ef41f9745e46789547c0a7fbe35bed` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:30.184805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94ef41f9745e46789547c0a7fbe35bed)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:30.186697 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:30.232315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94ef41f9745e46789547c0a7fbe35bed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.240486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94ef41f9745e46789547c0a7fbe35bed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.252962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94ef41f9745e46789547c0a7fbe35bed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.253606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94ef41f9745e46789547c0a7fbe35bed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.260447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94ef41f9745e46789547c0a7fbe35bed'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.266313 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1355/2691] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:30 2022] DELETE /identity/v3/users/94ef41f9745e46789547c0a7fbe35bed => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:30.270844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:30.271432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:30.271627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:30.290735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:30.291981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.301558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:30.363541 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1337/2692] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:30.373509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:30.375196 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.440205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce661948-10c9-4372-b9a9-21c9f88b8999 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:30.444111 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:30.444712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:30.444985 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:30.445254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:30.454272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:30.456740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:30.470978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1356/2693] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:30.491751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:30.493157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.553083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e74b87-662d-49f4-bdf6-221bb5440ae7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:30.556792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:30.557401 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:30.557678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:30.557950 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:30.559673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.626575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:30.628322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f359d6d-f0a7-4481-9e44-a4084a6b63f5 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:30.639459 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1338/2694] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:30.739047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:30.740613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.758331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502477c5-3eac-4590-b1b5-2591ea6dc9a5 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:30.763687 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:30.764889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:30.765170 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:30.765700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] PATH_INFO: `/v3/projects/ee36270279f94439a30b92a01c139495` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:30.767410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee36270279f94439a30b92a01c139495)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:30.769329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:30.811820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:30.824267 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee36270279f94439a30b92a01c139495', 'request_id': 'req-37a700f5-ddda-48e3-9a95-a3de33b23959'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.832145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee36270279f94439a30b92a01c139495', 'request_id': 'req-37a700f5-ddda-48e3-9a95-a3de33b23959'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.838590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee36270279f94439a30b92a01c139495', 'request_id': 'req-37a700f5-ddda-48e3-9a95-a3de33b23959'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.851441 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] The token cache is being invalidate because project ee36270279f94439a30b92a01c139495 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:30.851699 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.852492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-37a700f5-ddda-48e3-9a95-a3de33b23959 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:30.854065 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1357/2695] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:30 2022] DELETE /identity/v3/projects/ee36270279f94439a30b92a01c139495 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:30.859144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:30.859430 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:30.859707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:30.880494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:30.882051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:30.945618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1339/2696] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:31.012384 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.012602 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.012858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.034320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:31.035565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:31.047342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1358/2697] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:31.051690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.063166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1cd9cb-9da9-4fd6-bac5-e780d08edacd None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.070215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.070845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.071157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.071434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.072092 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:31.072395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.073978 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.081686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:31.088752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1340/2698] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:31.093438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.114241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1a9a39-d7a3-4fb4-80a8-96660c31b90d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.120360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.121106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.121569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.122254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.123123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.125112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.159402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1359/2699] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:31 2022] POST /identity/v3/projects => generated 397 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:31.163582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.174133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768875c3-a28d-445a-9437-1d41897b602d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.238838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.239725 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.240010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.240283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.240918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.242861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.264762 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.265947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1341/2700] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:31 2022] POST /identity/v3/users => generated 299 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:31.267429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:31.270568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.280818 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4dfb0-c6d5-433c-ad4c-b88581717912 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.285707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.286330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.286610 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.286883 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.287560 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.289184 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.299708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1342/2701] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:31.303550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.313330 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326b9c5-272d-44b3-8fc3-7a438400b8a1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.319134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.319768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.320001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.320505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.320927 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.322618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.327188 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd25ce1f-b4a9-4b67-bc04-51d93e72ffed admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.331528 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.332126 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.332410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.332732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.334021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1343/2702] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:31.334550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:31.337860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.347678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a0774a-39e5-44a2-bb82-dd096ce57931 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.353189 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcf3c3f-8ff8-4a41-b6ef-4a5cd1e92039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.353947 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcf3c3f-8ff8-4a41-b6ef-4a5cd1e92039 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.354408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcf3c3f-8ff8-4a41-b6ef-4a5cd1e92039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.354661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcf3c3f-8ff8-4a41-b6ef-4a5cd1e92039 admin admin] PATH_INFO: `/v3/projects/3105f3b64a9b41d6b84e095eeddf14a4/users/fe1b7a1c1a3049bbb03d6852fcf8eb07/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.370539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcf3c3f-8ff8-4a41-b6ef-4a5cd1e92039 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3105f3b64a9b41d6b84e095eeddf14a4, user_id=fe1b7a1c1a3049bbb03d6852fcf8eb07, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.373251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcf3c3f-8ff8-4a41-b6ef-4a5cd1e92039 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.387379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1344/2703] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:31 2022] PUT /identity/v3/projects/3105f3b64a9b41d6b84e095eeddf14a4/users/fe1b7a1c1a3049bbb03d6852fcf8eb07/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:31.398431 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.398731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.399045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.403566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.405042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.415021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1360/2704] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:31 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:31.430675 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] MFA Rules not processed for user `fe1b7a1c1a3049bbb03d6852fcf8eb07`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:31.432053 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:31.503743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1345/2705] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:31 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:31.515508 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:31.532481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb645b6-73f9-4ee8-91a4-ec67a8b21f7e service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:31.574488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:31.583443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:31.584194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:31.584534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:31.584982 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:31.593976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:31.595432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:31.604129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1361/2706] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:31 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:32.261986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:32.263463 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:32.297616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:32.321825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dd7689-2893-4ae5-a07b-0f32cb0ea588 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:32.325408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:32.326004 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:32.326280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:32.326550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:32.328280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:32.390658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:32.393186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:32.403678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1346/2707] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:35.575430 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:35.577869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:35.595200 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01109e83-0a31-46c1-9728-b02fbdc3a79f service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:35.600286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:35.601176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:35.601653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:35.601883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] PATH_INFO: `/v3/users/2decd678421e4231bb3b58cd57d600bc` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:35.603345 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2decd678421e4231bb3b58cd57d600bc)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:35.605239 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:35.651420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2decd678421e4231bb3b58cd57d600bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.658581 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2decd678421e4231bb3b58cd57d600bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.671567 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2decd678421e4231bb3b58cd57d600bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.672232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2decd678421e4231bb3b58cd57d600bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.680039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-86bcaffd-b25a-429b-bfe7-d5bba7516826 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2decd678421e4231bb3b58cd57d600bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.686237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1362/2708] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:35 2022] DELETE /identity/v3/users/2decd678421e4231bb3b58cd57d600bc => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:35.862304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:35.872784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89afc050-60df-4947-b667-16057d2fd7b1 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:35.931073 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:35.931705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:35.931987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:35.932253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] PATH_INFO: `/v3/projects/91bb02023bea4020ba1066ba4c85ff4d` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:35.933364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91bb02023bea4020ba1066ba4c85ff4d)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:35.935298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:35.981491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91bb02023bea4020ba1066ba4c85ff4d', 'request_id': 'req-17329c9b-235a-408e-8894-32a773cb5407'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.988839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91bb02023bea4020ba1066ba4c85ff4d', 'request_id': 'req-17329c9b-235a-408e-8894-32a773cb5407'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:35.994673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91bb02023bea4020ba1066ba4c85ff4d', 'request_id': 'req-17329c9b-235a-408e-8894-32a773cb5407'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:36.005769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] The token cache is being invalidate because project 91bb02023bea4020ba1066ba4c85ff4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:36.006057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:36.006671 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17329c9b-235a-408e-8894-32a773cb5407 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:36.008116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1347/2709] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:35 2022] DELETE /identity/v3/projects/91bb02023bea4020ba1066ba4c85ff4d => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:36.013202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.013489 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.013765 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.034619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:36.035836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:36.093556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1363/2710] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:36.101715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.102021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.102281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.122489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:36.123837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:36.132622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1348/2711] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:36.136971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.146403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19feba90-85aa-4f4d-ab92-833d270b5157 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.150858 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.151518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.151799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.152070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.152707 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:36.153010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.154577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.165892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1364/2712] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:36.170055 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.176040 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:36.179903 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463269c-ee0c-4509-b574-f4d48f5d14f1 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.185813 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.186497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.186999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.187444 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.188206 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.190956 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.217096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1349/2713] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:36 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:36.221132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.230896 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cc02c0-0625-41e2-932a-968864f8b15a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.274354 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:36.289149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.289684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.289959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.290226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.290846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.292754 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.313681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1365/2714] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:36 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:36.318083 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.328588 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa1122-9f07-453b-9aa6-5a4411c2ef6f admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.333348 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.333940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.334214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.334481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.335147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.336792 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.347124 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1350/2715] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:36.350856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.361782 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ce0bfc-dffd-4f34-8ee9-1c48a0e0eaa0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.367226 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c5c26a-c220-4387-b277-cb05c50b9449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.367992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c5c26a-c220-4387-b277-cb05c50b9449 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.368269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c5c26a-c220-4387-b277-cb05c50b9449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.368638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c5c26a-c220-4387-b277-cb05c50b9449 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.369162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c5c26a-c220-4387-b277-cb05c50b9449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.370763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c5c26a-c220-4387-b277-cb05c50b9449 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.381956 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1366/2716] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:36.386352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.396760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3b153-8a9a-4913-876c-6b0b966abafa admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.401662 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.402311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.402641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.402858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] PATH_INFO: `/v3/projects/0546fa3da432440fa932e5dafced75a7/users/c5ad0cf3d7c141ecaaab277b748ea1e2/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.422239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0546fa3da432440fa932e5dafced75a7, user_id=c5ad0cf3d7c141ecaaab277b748ea1e2, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.425601 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.441895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1351/2717] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:36 2022] PUT /identity/v3/projects/0546fa3da432440fa932e5dafced75a7/users/c5ad0cf3d7c141ecaaab277b748ea1e2/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:36.452801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.453098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.453405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.488973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] MFA Rules not processed for user `c5ad0cf3d7c141ecaaab277b748ea1e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:36.490225 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:36.562110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1367/2718] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:36.573612 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:36.575335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:36.649728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856dc31-e0d7-4e00-b2a1-e6cbe1d2e387 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:36.655393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6ea1b0-cdfe-4ea5-9769-872598fd27b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:36.655986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ea1b0-cdfe-4ea5-9769-872598fd27b6 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:36.656264 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ea1b0-cdfe-4ea5-9769-872598fd27b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:36.656556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ea1b0-cdfe-4ea5-9769-872598fd27b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:36.665673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ea1b0-cdfe-4ea5-9769-872598fd27b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:36.667327 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ea1b0-cdfe-4ea5-9769-872598fd27b6 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:36.676333 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1352/2719] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:37.538920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:37.540455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:37.607400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed9d26-e478-4e6e-bcaa-50c309d6a36a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:37.615410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:37.616084 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:37.616366 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:37.616638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] PATH_INFO: `/v3/users/59667bb9f8be401fab0d5a9fcd16ff7b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:37.617771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59667bb9f8be401fab0d5a9fcd16ff7b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:37.619727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:37.668816 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59667bb9f8be401fab0d5a9fcd16ff7b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:37.676223 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59667bb9f8be401fab0d5a9fcd16ff7b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:37.678398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:37.691824 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59667bb9f8be401fab0d5a9fcd16ff7b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:37.692501 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59667bb9f8be401fab0d5a9fcd16ff7b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:37.699353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59667bb9f8be401fab0d5a9fcd16ff7b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:37.705271 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1368/2720] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:37 2022] DELETE /identity/v3/users/59667bb9f8be401fab0d5a9fcd16ff7b => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:37.709782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:37.710262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:37.710482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:37.729465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:37.730608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:37.788746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1353/2721] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:37.798993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:37.800447 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:37.858175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95510b2-ba98-424e-bcea-9b22adf9ab71 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:37.863230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:37.864146 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:37.864633 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:37.865107 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:37.876247 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:37.877666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:37.887403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1369/2722] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:38.165893 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:38.167520 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:38.184432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a66c7ee-9bdf-46b5-a63d-71e091554423 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:38.190697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:38.192088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:38.192548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:38.193020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] PATH_INFO: `/v3/projects/9f711a760ebb4f7486df1deb767373c7` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:38.194728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f711a760ebb4f7486df1deb767373c7)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:38.197379 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:38.245042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:38.269539 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f711a760ebb4f7486df1deb767373c7', 'request_id': 'req-92056db2-4aea-4011-93f7-8136ec70aac3'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.276536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f711a760ebb4f7486df1deb767373c7', 'request_id': 'req-92056db2-4aea-4011-93f7-8136ec70aac3'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.283194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f711a760ebb4f7486df1deb767373c7', 'request_id': 'req-92056db2-4aea-4011-93f7-8136ec70aac3'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.294698 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] The token cache is being invalidate because project 9f711a760ebb4f7486df1deb767373c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:38.295165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.295881 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.297489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1354/2723] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:38 2022] DELETE /identity/v3/projects/9f711a760ebb4f7486df1deb767373c7 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:38.301837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:38.315992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6671797e-de51-4ae3-a2b0-cb9ecbdaaaf9 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:38.384060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:38.384691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:38.384969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:38.385272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] PATH_INFO: `/v3/users/4162d42730b64a65b7a96e3b5323c1d3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:38.386342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] RBAC: Authorizing `identity:delete_user(user_id=4162d42730b64a65b7a96e3b5323c1d3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:38.388494 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:38.442590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4162d42730b64a65b7a96e3b5323c1d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.450159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4162d42730b64a65b7a96e3b5323c1d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.462442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4162d42730b64a65b7a96e3b5323c1d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.463149 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4162d42730b64a65b7a96e3b5323c1d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.468431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:38.471262 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4162d42730b64a65b7a96e3b5323c1d3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.478618 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1370/2724] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:38 2022] DELETE /identity/v3/users/4162d42730b64a65b7a96e3b5323c1d3 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:38.694354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:38.695906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:38.709046 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92056db2-4aea-4011-93f7-8136ec70aac3 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:38.747558 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:38.750156 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:38.771872 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:38.772771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:38.773089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:38.773500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] PATH_INFO: `/v3/users/fe1b7a1c1a3049bbb03d6852fcf8eb07` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:38.774742 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe1b7a1c1a3049bbb03d6852fcf8eb07)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:38.776796 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:38.822915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dee678f-cdaf-46da-ba78-472f470d760a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:38.823684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe1b7a1c1a3049bbb03d6852fcf8eb07'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.828280 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:38.828914 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:38.829791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:38.830061 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] PATH_INFO: `/v3/projects/df202069e3b1467c940f64d4b8b89904` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:38.831206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] RBAC: Authorizing `identity:delete_project(project_id=df202069e3b1467c940f64d4b8b89904)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:38.832750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe1b7a1c1a3049bbb03d6852fcf8eb07'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.833094 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:38.847678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe1b7a1c1a3049bbb03d6852fcf8eb07'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.848362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe1b7a1c1a3049bbb03d6852fcf8eb07'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.855314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-9bdcb158-c388-4902-8faf-0ad66e6315bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe1b7a1c1a3049bbb03d6852fcf8eb07'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.861301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1355/2725] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:38 2022] DELETE /identity/v3/users/fe1b7a1c1a3049bbb03d6852fcf8eb07 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:38.866047 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:38.866472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:38.866816 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:38.881983 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df202069e3b1467c940f64d4b8b89904', 'request_id': 'req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.885906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:38.887233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:38.890057 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df202069e3b1467c940f64d4b8b89904', 'request_id': 'req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.896904 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df202069e3b1467c940f64d4b8b89904', 'request_id': 'req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.908308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] The token cache is being invalidate because project df202069e3b1467c940f64d4b8b89904 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:38.908606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.909298 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:38.910795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1371/2726] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:38 2022] DELETE /identity/v3/projects/df202069e3b1467c940f64d4b8b89904 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:38.914826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:38.916127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:38.917745 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:38.951010 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1356/2727] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:38.960721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:38.962066 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:38.980583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e29b16-e9f0-42a8-87d1-5a8982c5a33f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:38.985895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:38.986540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:38.986800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:38.987102 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] PATH_INFO: `/v3/users/66daa5467c7b46a8a8b647244c9a3675` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:38.988241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] RBAC: Authorizing `identity:delete_user(user_id=66daa5467c7b46a8a8b647244c9a3675)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:38.990131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.019621 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72eb733-1ff8-47e4-8aa5-8916fdebd4a1 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.020215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:39.024029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.024705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.025011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.025314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.035100 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.036641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.043640 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '66daa5467c7b46a8a8b647244c9a3675'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.046705 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1357/2728] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:39.052874 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '66daa5467c7b46a8a8b647244c9a3675'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.066269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '66daa5467c7b46a8a8b647244c9a3675'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.067278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '66daa5467c7b46a8a8b647244c9a3675'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.074230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '66daa5467c7b46a8a8b647244c9a3675'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.080527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1372/2729] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:38 2022] DELETE /identity/v3/users/66daa5467c7b46a8a8b647244c9a3675 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:39.421265 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.422710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:39.429753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.431461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:39.439247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e35c09-195d-41c9-8d2c-a989e58175a6 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.506625 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.507295 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.507579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.507853 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] PATH_INFO: `/v3/projects/3105f3b64a9b41d6b84e095eeddf14a4` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.508984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3105f3b64a9b41d6b84e095eeddf14a4)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.510887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.529873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57a32c0-38db-4e09-9723-7de51624354d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.535046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.535607 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.535885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.536155 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] PATH_INFO: `/v3/projects/953cfadb5c4a4d2b8df7dc4604b6cd6d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.537405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] RBAC: Authorizing `identity:delete_project(project_id=953cfadb5c4a4d2b8df7dc4604b6cd6d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.539629 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.559113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3105f3b64a9b41d6b84e095eeddf14a4', 'request_id': 'req-d4876795-732f-4095-8811-010e878f4108'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.567434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3105f3b64a9b41d6b84e095eeddf14a4', 'request_id': 'req-d4876795-732f-4095-8811-010e878f4108'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.573514 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3105f3b64a9b41d6b84e095eeddf14a4', 'request_id': 'req-d4876795-732f-4095-8811-010e878f4108'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.584591 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] The token cache is being invalidate because project 3105f3b64a9b41d6b84e095eeddf14a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:39.584884 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.585503 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-d4876795-732f-4095-8811-010e878f4108 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.587017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1358/2730] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:39 2022] DELETE /identity/v3/projects/3105f3b64a9b41d6b84e095eeddf14a4 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:39.591692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697552d-6757-42f1-843a-1bc651c46b63 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.591975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697552d-6757-42f1-843a-1bc651c46b63 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.592080 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '953cfadb5c4a4d2b8df7dc4604b6cd6d', 'request_id': 'req-16cba162-787c-4fe8-ac3c-a6a49e852379'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.592283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697552d-6757-42f1-843a-1bc651c46b63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.602284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '953cfadb5c4a4d2b8df7dc4604b6cd6d', 'request_id': 'req-16cba162-787c-4fe8-ac3c-a6a49e852379'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.608183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '953cfadb5c4a4d2b8df7dc4604b6cd6d', 'request_id': 'req-16cba162-787c-4fe8-ac3c-a6a49e852379'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.612649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3697552d-6757-42f1-843a-1bc651c46b63 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:39.613864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3697552d-6757-42f1-843a-1bc651c46b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:39.618417 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:39.620125 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] The token cache is being invalidate because project 953cfadb5c4a4d2b8df7dc4604b6cd6d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:39.620412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.621054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-16cba162-787c-4fe8-ac3c-a6a49e852379 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:39.622566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1373/2731] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:39 2022] DELETE /identity/v3/projects/953cfadb5c4a4d2b8df7dc4604b6cd6d => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:39.627940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe844c3-1d9e-4da8-b981-d6ce68aabf8d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.628443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe844c3-1d9e-4da8-b981-d6ce68aabf8d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.628883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe844c3-1d9e-4da8-b981-d6ce68aabf8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.650693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4fe844c3-1d9e-4da8-b981-d6ce68aabf8d None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:39.651963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4fe844c3-1d9e-4da8-b981-d6ce68aabf8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:39.683891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1359/2732] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:39.686898 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:39.693552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.693836 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.694139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.713867 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:39.715109 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:39.718343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1374/2733] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:39.723687 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1360/2734] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:39.727694 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.727835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.728121 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.728468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.737104 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecbbd80-5b6f-4047-a8a4-41a88442b1d7 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.742205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.742838 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.743146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.743419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.744059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:39.744358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.745946 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.748968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:39.750162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:39.757861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1361/2735] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:46:39.758793 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1375/2736] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:39.762467 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.762614 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.772369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6128d335-6929-4582-8aec-1dce02d34a53 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.772721 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0156a78f-c142-4aba-94f8-c4245298c4eb admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.776974 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.777320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.777559 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.777840 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.777940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.778113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.778223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.778415 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.778751 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:39.779070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.779185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.780725 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.780943 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.793584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1376/2737] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:46:39.797906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.808799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1362/2738] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:39.808977 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a714e-c65b-4b4e-9afe-03ebbaa031e8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.813277 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.866548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.867216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.867493 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.867767 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.868403 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.870335 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.876342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccd0943-980a-4f9b-9d50-06947b15b4dd admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.881034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.881647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.881910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.882182 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.882810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.884788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.898518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1377/2739] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/projects => generated 377 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:39.902670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.912187 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1363/2740] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:39.913205 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:39.923738 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355e620-3f00-4855-bce9-8a2aed294533 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.964587 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0165ef-31bb-468c-b803-2938adbb751f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:39.969370 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.970709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.970857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.971154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.971892 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.973783 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.974912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:39.982468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:39.983496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:39.984158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:39.984434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:39.985577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:39.987462 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:39.995692 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1378/2741] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:39 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:40.000036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.001312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1364/2742] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:39 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:40.005302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.009795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07914986-6647-4ea8-b7ec-5ce89df3de5a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.014464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.014910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1031f7c-66da-473f-869a-139aff363fc1 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.015105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.015385 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.015679 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.016361 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.018406 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.020115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.020693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.020972 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.021247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.021870 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.023944 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.028732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1379/2743] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:40.033103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.040315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1365/2744] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:40.041389 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.042711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5fdb7a-8340-463f-baf5-3ea717d07b8e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.047730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.048259 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.048544 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.048823 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.049460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.051145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.051273 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadef716-026d-4fd0-a850-074662dfdf01 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.055955 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43664eeb-5467-450c-8296-b05821bab306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.056597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43664eeb-5467-450c-8296-b05821bab306 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.056857 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43664eeb-5467-450c-8296-b05821bab306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.057131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43664eeb-5467-450c-8296-b05821bab306 admin admin] PATH_INFO: `/v3/projects/9931b1bca2464db7bcb90b91567cab28/users/c163b2e8db43490196e93b5fe4d45378/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.061881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1380/2745] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:46:40.066151 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.072651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43664eeb-5467-450c-8296-b05821bab306 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9931b1bca2464db7bcb90b91567cab28, user_id=c163b2e8db43490196e93b5fe4d45378, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.075518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43664eeb-5467-450c-8296-b05821bab306 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.075916 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c5c0b-25ba-4941-b8e0-4009b8078e44 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.081409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984e1d0-2b0d-4e58-8ea8-fb6543d2acfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.082072 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984e1d0-2b0d-4e58-8ea8-fb6543d2acfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.082328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984e1d0-2b0d-4e58-8ea8-fb6543d2acfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.082601 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984e1d0-2b0d-4e58-8ea8-fb6543d2acfb admin admin] PATH_INFO: `/v3/projects/ea4c285aae544f47a91407ff79414128/users/9ece0d7befa845c0ab2a33bfb10b7850/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.085743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:40.089486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1366/2746] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:40 2022] PUT /identity/v3/projects/9931b1bca2464db7bcb90b91567cab28/users/c163b2e8db43490196e93b5fe4d45378/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:40.094354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.094999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.095315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.100471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984e1d0-2b0d-4e58-8ea8-fb6543d2acfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea4c285aae544f47a91407ff79414128, user_id=9ece0d7befa845c0ab2a33bfb10b7850, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.103315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984e1d0-2b0d-4e58-8ea8-fb6543d2acfb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.114686 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:40.116020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1381/2747] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:40 2022] PUT /identity/v3/projects/ea4c285aae544f47a91407ff79414128/users/9ece0d7befa845c0ab2a33bfb10b7850/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:40.116179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:40.120928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.121212 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.121492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.140320 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:40.141565 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:40.179767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1367/2748] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:40.190238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.192001 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:40.200571 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1382/2749] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:46:40.210154 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:40.211990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:40.253367 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0abccf-0ac9-45de-8538-726b4704f650 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.261918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.262960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.262960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.263150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.271360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e555c6e-f330-4ab9-8d31-d32149f58419 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:40.274080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.275171 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba859aff-1dd0-49ae-9511-e7ba86c566f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:40.275577 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.277711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba859aff-1dd0-49ae-9511-e7ba86c566f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:40.280941 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba859aff-1dd0-49ae-9511-e7ba86c566f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:40.281232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba859aff-1dd0-49ae-9511-e7ba86c566f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:40.286443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1368/2750] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:40.290973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba859aff-1dd0-49ae-9511-e7ba86c566f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:40.292417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba859aff-1dd0-49ae-9511-e7ba86c566f4 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:40.302129 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1383/2751] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:46:44.314969 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:44.317424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:44.342645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38f2a6-bbde-48bd-a1df-df32c209d9b6 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:44.359607 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:44.360572 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:44.361030 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:44.361479 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] PATH_INFO: `/v3/users/c5ad0cf3d7c141ecaaab277b748ea1e2` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:44.363281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5ad0cf3d7c141ecaaab277b748ea1e2)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:44.366105 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:44.477497 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5ad0cf3d7c141ecaaab277b748ea1e2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:44.495076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5ad0cf3d7c141ecaaab277b748ea1e2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:44.497732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:44.519994 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5ad0cf3d7c141ecaaab277b748ea1e2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:44.520743 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5ad0cf3d7c141ecaaab277b748ea1e2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:44.528732 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5ad0cf3d7c141ecaaab277b748ea1e2'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:44.536453 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1369/2752] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:46:44 2022] DELETE /identity/v3/users/c5ad0cf3d7c141ecaaab277b748ea1e2 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:44.541680 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb29704b-284d-4c79-87d9-1e52ed60acbf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:44.542180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb29704b-284d-4c79-87d9-1e52ed60acbf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:44.542281 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb29704b-284d-4c79-87d9-1e52ed60acbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:44.575875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-eb29704b-284d-4c79-87d9-1e52ed60acbf None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:44.580287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eb29704b-284d-4c79-87d9-1e52ed60acbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:44.744704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1384/2753] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:44.828771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:44.831957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:45.006923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bbfae-ac49-4096-8bb5-dfca58d4a4ad admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:45.016937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:45.017829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:45.018163 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:45.018442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:45.052968 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:45.054844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:45.084593 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1370/2754] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:45.724284 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:45.725740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:45.726446 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:45.853832 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] MFA Rules not processed for user `9ece0d7befa845c0ab2a33bfb10b7850`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:45.856110 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:45.899152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:45.981708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1385/2755] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:45.985864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:45.988345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.003580 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5633501-4710-484b-9cf7-5edcfe360e17 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.009274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.010098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.010548 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.011041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.011874 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.014301 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.074728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1371/2756] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:45 2022] POST /identity/v3/projects => generated 379 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:46.076039 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.090290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dc0175-cdb8-4bc8-9f70-6ba03079d75d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.104715 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.107259 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.189498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.190252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.190591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.191017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.191921 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.197794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.254873 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1386/2757] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/users => generated 296 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:46.262744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.275584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd7a94-7188-46a7-b89b-3304a96dd023 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.291196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac28190-b5e6-4f14-a9ce-bd6aa4a024b9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.296064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.296828 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.297160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.297498 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.298442 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.300158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.307005 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.307678 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.307958 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.308233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] PATH_INFO: `/v3/projects/0546fa3da432440fa932e5dafced75a7` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.309551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0546fa3da432440fa932e5dafced75a7)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.311641 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.336351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1387/2758] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:46 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:46.337355 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.356664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c136e-6a9c-40ac-b6f0-251a58981cd4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.362210 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.363206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.363557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.364063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] PATH_INFO: `/v3/projects/c2ef1a079e644830876e5d6423987db9/users/611d6e38f9d9486f8d95d2ca2d692819/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.386362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2ef1a079e644830876e5d6423987db9, user_id=611d6e38f9d9486f8d95d2ca2d692819, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.389681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.414091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1388/2759] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:46 2022] PUT /identity/v3/projects/c2ef1a079e644830876e5d6423987db9/users/611d6e38f9d9486f8d95d2ca2d692819/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:46.418486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.445443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0546fa3da432440fa932e5dafced75a7', 'request_id': 'req-a718e021-cc94-4150-81e3-d3efcb3da565'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:46.459547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0546fa3da432440fa932e5dafced75a7', 'request_id': 'req-a718e021-cc94-4150-81e3-d3efcb3da565'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:46.461851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:46.486988 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0546fa3da432440fa932e5dafced75a7', 'request_id': 'req-a718e021-cc94-4150-81e3-d3efcb3da565'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:46.504494 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] The token cache is being invalidate because project 0546fa3da432440fa932e5dafced75a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:46:46.505003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:46.506029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-a718e021-cc94-4150-81e3-d3efcb3da565 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:46:46.511729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1372/2760] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:46:46 2022] DELETE /identity/v3/projects/0546fa3da432440fa932e5dafced75a7 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:46.513125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.513412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.513711 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.534733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:46.536018 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.551217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f681fedc-989b-4681-8e64-06c498e2e307 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.557619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465dc1af-d79c-4339-8c5e-dfc6dad6a945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.558626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465dc1af-d79c-4339-8c5e-dfc6dad6a945 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.559050 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465dc1af-d79c-4339-8c5e-dfc6dad6a945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.559424 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465dc1af-d79c-4339-8c5e-dfc6dad6a945 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.561022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465dc1af-d79c-4339-8c5e-dfc6dad6a945 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.565287 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465dc1af-d79c-4339-8c5e-dfc6dad6a945 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.577812 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1389/2761] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:46 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:46.579165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.579454 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.579730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.602973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:46.619537 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] MFA Rules not processed for user `c163b2e8db43490196e93b5fe4d45378`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:46.620768 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.631249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1373/2762] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:46.632085 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.640210 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.653581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5987615-d7a4-48ef-a907-8ca88b2bce0a None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.660700 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab1fb52-40f5-46b4-a8a0-46f067ef04a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.661602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab1fb52-40f5-46b4-a8a0-46f067ef04a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.661957 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab1fb52-40f5-46b4-a8a0-46f067ef04a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.662299 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab1fb52-40f5-46b4-a8a0-46f067ef04a2 admin admin] PATH_INFO: `/v3/projects/c2ef1a079e644830876e5d6423987db9/users/611d6e38f9d9486f8d95d2ca2d692819/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.664302 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab1fb52-40f5-46b4-a8a0-46f067ef04a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2ef1a079e644830876e5d6423987db9, user_id=611d6e38f9d9486f8d95d2ca2d692819, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.667976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab1fb52-40f5-46b4-a8a0-46f067ef04a2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.682227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1374/2763] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:46 2022] PUT /identity/v3/projects/c2ef1a079e644830876e5d6423987db9/users/611d6e38f9d9486f8d95d2ca2d692819/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:46.683525 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.683812 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.684086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.712486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:46.714419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.721254 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1390/2764] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:46.727344 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.730010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:46.750031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467a4ef8-1600-4879-8114-e0fa3d0b5c5c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.830382 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.831311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.831747 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.832192 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.833009 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.835429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.844127 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1375/2765] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:46.847507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.863684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9f967-597b-4301-aa55-f14fc1dff041 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.871093 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.871767 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.872473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.872473 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.873111 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:46:46.873420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:46.876099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:46.879227 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1391/2766] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/projects => generated 385 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:46.884002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.902677 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1376/2767] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:46:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:46:46.907835 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:46.920041 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a4cbf2-7e5f-4289-be64-cea4812d18c9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.984282 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814685d-c8b2-47d7-91a0-e0bab06f5936 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:46.992120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:46.992952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:46.993356 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:46.993943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:46.994827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.000912 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.022817 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.023558 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.023839 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.024130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.025225 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.027216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.033454 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:47.045864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1392/2768] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/users => generated 299 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:47.050173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.077514 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb95fa-0f9d-4754-b875-21044bfce8d4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.084250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1377/2769] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:46 2022] POST /identity/v3/projects => generated 365 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:46:47.087513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.088032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.088306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.088420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.088578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.089230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.093883 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.114712 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1393/2770] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:47 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:47.121318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.133773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:47.145433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f192fb-6005-438c-937d-64bbe4674816 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.179410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc624c6c-ddf6-4a1b-aef0-84de49fbada8 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.184734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.185550 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.185933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.186316 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.187346 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.189722 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.239984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.240947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.241794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.244160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] PATH_INFO: `/v3/projects/045033350adb4269aac1665b7ff23237/users/dc1ff8a54ae34493bc88cb3ca209f2fa/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.251027 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1378/2771] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:47 2022] POST /identity/v3/users => generated 283 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:47.264845 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.268415 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=045033350adb4269aac1665b7ff23237, user_id=dc1ff8a54ae34493bc88cb3ca209f2fa, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.271387 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.277912 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15d784a-6661-4f37-8fd0-d7aae1f6d8a3 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.284846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.285536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.285879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.286215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.286923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.289674 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1394/2772] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:47 2022] PUT /identity/v3/projects/045033350adb4269aac1665b7ff23237/users/dc1ff8a54ae34493bc88cb3ca209f2fa/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:47.294651 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.300759 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.341076 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1379/2773] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:47 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:47.342896 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.373384 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9b86c4-35cc-4cd6-8008-3ee67b5c8172 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.449645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2818e4c5-6c7c-4e13-8018-2d779173b9a9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.458773 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.460113 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.460404 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.460684 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.461486 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.463452 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.483402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1395/2774] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:47 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:47.484457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.513881 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e939ca2-f1fa-46cb-adae-a42d0ee20939 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.521231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.521937 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.522222 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.522529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.523217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.524876 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.527477 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a62ae-c97d-43b9-87db-49f6de3a483b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.528583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a62ae-c97d-43b9-87db-49f6de3a483b admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.529073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a62ae-c97d-43b9-87db-49f6de3a483b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.530014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a62ae-c97d-43b9-87db-49f6de3a483b admin admin] PATH_INFO: `/v3/projects/045033350adb4269aac1665b7ff23237/users/dc1ff8a54ae34493bc88cb3ca209f2fa/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.534290 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7a62ae-c97d-43b9-87db-49f6de3a483b admin admin] RBAC: Authorizing `identity:create_grant(project_id=045033350adb4269aac1665b7ff23237, user_id=dc1ff8a54ae34493bc88cb3ca209f2fa, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.540067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7a62ae-c97d-43b9-87db-49f6de3a483b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.545253 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1380/2775] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:47 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:47.550319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.560025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1396/2776] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:47 2022] PUT /identity/v3/projects/045033350adb4269aac1665b7ff23237/users/dc1ff8a54ae34493bc88cb3ca209f2fa/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:47.572930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c5c351-74cd-4eec-99ae-b42c9b4f823a admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.667940 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.668684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.669051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.669762 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] PATH_INFO: `/v3/projects/730201815e6d4f45b7fff4e3da88cec5/users/21344468b104403886b65cd626c776a5/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.675622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:47.696275 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] RBAC: Authorizing `identity:create_grant(project_id=730201815e6d4f45b7fff4e3da88cec5, user_id=21344468b104403886b65cd626c776a5, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.699447 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:47.730534 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1381/2777] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:47 2022] PUT /identity/v3/projects/730201815e6d4f45b7fff4e3da88cec5/users/21344468b104403886b65cd626c776a5/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:47.732297 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.732778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.733011 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.779089 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] MFA Rules not processed for user `21344468b104403886b65cd626c776a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:47.780791 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:47.876604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:47.897035 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1397/2778] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:47 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:47.902043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:47.978250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f9353-f1c4-4549-89c3-ed8db334854d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:47.983709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:47.984385 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:47.984693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:47.984964 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:47.985638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:47.987694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.015011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1382/2779] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:46:47 2022] POST /identity/v3/projects => generated 365 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:48.020533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:48.037123 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69c2b6a-9e97-4f55-8453-15d32d013513 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:48.113578 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:48.114426 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.114972 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.115329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.116175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:48.118376 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.198748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1398/2780] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:46:48 2022] POST /identity/v3/users => generated 289 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:46:48.202716 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:48.215997 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b81b64-dc40-44ff-9f41-d1763e30b5b7 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:48.233966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:48.235524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.235984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.236405 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.237086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:48.238726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.257350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1383/2781] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:48 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:48.261653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:48.284906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f37fa66-8d41-44d4-8d80-b7d3a820cd8c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:48.301019 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:48.302095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.302740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.305325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] PATH_INFO: `/v3/projects/69220046c74b47278f2481ef576b21fa/users/6bc8cc03d7e647c2a93b340b761108eb/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.335079 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=69220046c74b47278f2481ef576b21fa, user_id=6bc8cc03d7e647c2a93b340b761108eb, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:48.338689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.358549 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1399/2782] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:48 2022] PUT /identity/v3/projects/69220046c74b47278f2481ef576b21fa/users/6bc8cc03d7e647c2a93b340b761108eb/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:46:48.359885 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:48.374408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd15e5b9-8593-4aaf-b63a-28af8ab02dff admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:48.484875 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbf082c-0926-4061-8dee-c8e0e04dcb4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:48.485551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf082c-0926-4061-8dee-c8e0e04dcb4f admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.485832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf082c-0926-4061-8dee-c8e0e04dcb4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.486101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf082c-0926-4061-8dee-c8e0e04dcb4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.486770 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbf082c-0926-4061-8dee-c8e0e04dcb4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:48.488645 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbf082c-0926-4061-8dee-c8e0e04dcb4f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.506413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1384/2783] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:46:48 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:46:48.512908 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:48.526548 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9546a4-5d63-4837-a840-25ef8d74ce0d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:48.533808 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:48.534519 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.534855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.535193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] PATH_INFO: `/v3/projects/69220046c74b47278f2481ef576b21fa/users/6bc8cc03d7e647c2a93b340b761108eb/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.537560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] RBAC: Authorizing `identity:create_grant(project_id=69220046c74b47278f2481ef576b21fa, user_id=6bc8cc03d7e647c2a93b340b761108eb, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:48.540530 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.564362 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1400/2784] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:46:48 2022] PUT /identity/v3/projects/69220046c74b47278f2481ef576b21fa/users/6bc8cc03d7e647c2a93b340b761108eb/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:46:48.565685 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.566156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.566688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.627667 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] MFA Rules not processed for user `6bc8cc03d7e647c2a93b340b761108eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:48.629343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:48.695032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:48.741014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1385/2785] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:48 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:48.752405 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:48.754235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:48.902584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc3121c-0862-426e-9f70-b07014156abf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:48.906771 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424c1dc9-20b0-4463-a0b3-91b5f6b44c5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:48.907527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424c1dc9-20b0-4463-a0b3-91b5f6b44c5c service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:48.907849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424c1dc9-20b0-4463-a0b3-91b5f6b44c5c service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:48.908116 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424c1dc9-20b0-4463-a0b3-91b5f6b44c5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:48.921988 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424c1dc9-20b0-4463-a0b3-91b5f6b44c5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:48.924492 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424c1dc9-20b0-4463-a0b3-91b5f6b44c5c service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:48.935889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1401/2786] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:48 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:49.065834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:49.076832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0333160c-1087-4b9f-8142-4925e306f4ff None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:49.085393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:49.086167 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:49.086553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:49.086954 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:49.240901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:49.242650 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:49.260137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1386/2787] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:49 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:54.900577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b15911-22d3-418a-8b51-188efe455ccd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:54.901106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b15911-22d3-418a-8b51-188efe455ccd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:54.901106 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b15911-22d3-418a-8b51-188efe455ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:54.941371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-33b15911-22d3-418a-8b51-188efe455ccd None None] MFA Rules not processed for user `611d6e38f9d9486f8d95d2ca2d692819`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:54.943278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-33b15911-22d3-418a-8b51-188efe455ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:54.992229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:55.040706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1402/2788] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:54 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:55.048771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:55.050298 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:55.168771 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda758af-b5e6-41da-bdab-622a4da4ee83 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:55.175516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:55.176145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:55.176428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:55.176704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:55.180458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:55.272967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:55.276411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:55.299568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1387/2789] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:55.483022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8a2cb-8080-4128-bdda-25d5edb2d260 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:55.483611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8a2cb-8080-4128-bdda-25d5edb2d260 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:55.484114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8a2cb-8080-4128-bdda-25d5edb2d260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:55.534596 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-44a8a2cb-8080-4128-bdda-25d5edb2d260 None None] MFA Rules not processed for user `dc1ff8a54ae34493bc88cb3ca209f2fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:55.536634 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-44a8a2cb-8080-4128-bdda-25d5edb2d260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:55.708800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1403/2790] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:46:55 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:55.720465 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:55.722096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:55.752139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:46:55.865279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690bebe-1816-43d8-90cf-7cdf18192bc9 service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:55.869779 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa7372a6f3ba4f3786becc1389147d2d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:55.871026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:55.871536 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:55.871975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:55.999628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:56.002086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:56.013436 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1388/2791] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:46:55 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:46:56.196794 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8881110-47bd-4663-a019-64560b3b615c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:56.197329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8881110-47bd-4663-a019-64560b3b615c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:56.197465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8881110-47bd-4663-a019-64560b3b615c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:56.226691 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f8881110-47bd-4663-a019-64560b3b615c None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:46:56.228483 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f8881110-47bd-4663-a019-64560b3b615c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:56.344748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1404/2792] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:46:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:46:56.406655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:46:56.408319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:46:56.514923 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c357544-a8f4-4670-80d8-c0feb0eabfcd service glance] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:46:56.521032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2aecba-abd4-4cfb-a917-268b62f0a604 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:46:56.521649 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2aecba-abd4-4cfb-a917-268b62f0a604 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:46:56.522086 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2aecba-abd4-4cfb-a917-268b62f0a604 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:46:56.522517 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2aecba-abd4-4cfb-a917-268b62f0a604 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:46:56.532414 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2aecba-abd4-4cfb-a917-268b62f0a604 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:46:56.533726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2aecba-abd4-4cfb-a917-268b62f0a604 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:46:56.544800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1389/2793] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:46:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:47:00.588257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8881110-47bd-4663-a019-64560b3b615c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:00.664821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:00.838209 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8881110-47bd-4663-a019-64560b3b615c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:00.848248 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:00.848527 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:00.848682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:00.848964 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] PATH_INFO: `/v3/users/21344468b104403886b65cd626c776a5` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:00.850216 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21344468b104403886b65cd626c776a5)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:00.852140 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:00.928820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21344468b104403886b65cd626c776a5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:00.943835 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21344468b104403886b65cd626c776a5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:00.961383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21344468b104403886b65cd626c776a5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:00.961819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21344468b104403886b65cd626c776a5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:00.970444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21344468b104403886b65cd626c776a5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:00.980842 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1405/2794] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:00 2022] DELETE /identity/v3/users/21344468b104403886b65cd626c776a5 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:00.982318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:00.982848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:00.982848 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:01.039442 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:01.041166 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:01.070760 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:01.142318 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1390/2795] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:01.154612 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:01.157161 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:01.241414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dae32db-ca93-4a39-a7c2-03463fbe86f8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:01.248393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:01.249451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:01.249894 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:01.250339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:01.264736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:01.273919 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:01.287993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1406/2796] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:01.652849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:01.654322 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:01.693973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c938f-6dd2-4628-b9cb-151ec4de08b9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:01.725150 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:01.727358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:01.727837 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:01.728224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] PATH_INFO: `/v3/projects/730201815e6d4f45b7fff4e3da88cec5` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:01.729576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] RBAC: Authorizing `identity:delete_project(project_id=730201815e6d4f45b7fff4e3da88cec5)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:01.731768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:01.818165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '730201815e6d4f45b7fff4e3da88cec5', 'request_id': 'req-4306f7d0-ae9b-460b-a8a5-357bb5b08411'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:01.832217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '730201815e6d4f45b7fff4e3da88cec5', 'request_id': 'req-4306f7d0-ae9b-460b-a8a5-357bb5b08411'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:01.840320 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '730201815e6d4f45b7fff4e3da88cec5', 'request_id': 'req-4306f7d0-ae9b-460b-a8a5-357bb5b08411'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:01.870478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] The token cache is being invalidate because project 730201815e6d4f45b7fff4e3da88cec5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:01.871289 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:01.873933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:01.874976 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1391/2797] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:01 2022] DELETE /identity/v3/projects/730201815e6d4f45b7fff4e3da88cec5 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:01.879294 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:01.894007 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212a2d8-a032-4a6d-8ffc-178ac9dc8d84 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:01.928306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:01.994615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:01.994615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:01.994615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:01.994615 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] PATH_INFO: `/v3/users/6bc8cc03d7e647c2a93b340b761108eb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:01.996351 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bc8cc03d7e647c2a93b340b761108eb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:01.998469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:02.067143 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bc8cc03d7e647c2a93b340b761108eb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.080991 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bc8cc03d7e647c2a93b340b761108eb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.095706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bc8cc03d7e647c2a93b340b761108eb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.096316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bc8cc03d7e647c2a93b340b761108eb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.104772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-08e6e53e-b3e0-4565-a4b7-33a1b647b397 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bc8cc03d7e647c2a93b340b761108eb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.114534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1407/2798] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:01 2022] DELETE /identity/v3/users/6bc8cc03d7e647c2a93b340b761108eb => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:02.489643 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:02.491371 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:02.527576 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4306f7d0-ae9b-460b-a8a5-357bb5b08411 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:02.625735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:02.626396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:02.626694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:02.627069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] PATH_INFO: `/v3/projects/69220046c74b47278f2481ef576b21fa` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:02.628507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69220046c74b47278f2481ef576b21fa)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:02.633017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:02.752516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:02.755981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69220046c74b47278f2481ef576b21fa', 'request_id': 'req-f9543569-61e2-442e-b236-f73704811e57'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.767131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69220046c74b47278f2481ef576b21fa', 'request_id': 'req-f9543569-61e2-442e-b236-f73704811e57'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.780070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69220046c74b47278f2481ef576b21fa', 'request_id': 'req-f9543569-61e2-442e-b236-f73704811e57'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.806045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] The token cache is being invalidate because project 69220046c74b47278f2481ef576b21fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:02.806045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.807014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f9543569-61e2-442e-b236-f73704811e57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:02.811247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1392/2799] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:02 2022] DELETE /identity/v3/projects/69220046c74b47278f2481ef576b21fa => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:02.816772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:02.817130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:02.817444 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:02.892906 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:02.894875 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:03.003258 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.003568 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.003832 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.008134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1408/2800] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:03.028930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:03.030197 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:03.042165 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1393/2801] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:03.046585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.066176 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa67df-f46d-4946-8e81-d1bd487b45cf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.072177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.072865 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.073153 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.073364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.074000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:47:03.074327 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.076082 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.087288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:03.101872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1409/2802] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:47:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:47:03.106726 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.121017 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26614f4-67d1-4ef8-b356-309856bf4545 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.135679 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.136315 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.136590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.136863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.137513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.139495 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.210456 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1394/2803] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:47:03 2022] POST /identity/v3/projects => generated 387 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:03.214661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.228202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac77750b-fe2f-4f75-b670-96eac80660e3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.311040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.311540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.311821 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.312095 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.313266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.315619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.352621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1410/2804] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:47:03 2022] POST /identity/v3/users => generated 294 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:47:03.356522 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.368949 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468cef2-f6bd-4161-b6e6-38c41dace283 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.380483 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.381133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.381443 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.381733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.382427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.384151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.407509 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.418254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1395/2805] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:03 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:47:03.419649 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c18f55-0cfc-498c-a22a-3c814832db4e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.430586 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42786d69-f82b-480f-aeac-34e9f712ae91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.431889 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42786d69-f82b-480f-aeac-34e9f712ae91 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.432337 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42786d69-f82b-480f-aeac-34e9f712ae91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.432888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42786d69-f82b-480f-aeac-34e9f712ae91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.433888 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42786d69-f82b-480f-aeac-34e9f712ae91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.435701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42786d69-f82b-480f-aeac-34e9f712ae91 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.456216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.465690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1411/2806] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:03 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:47:03.467039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4250be-be57-4278-94fb-037765da4d51 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.472392 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.473059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.473343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.473622 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] PATH_INFO: `/v3/projects/ecd21a4bb6494ff5b219037d559e0805/users/7e49cd16847c4a46afd83b66a9ca5ddd/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.490555 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecd21a4bb6494ff5b219037d559e0805, user_id=7e49cd16847c4a46afd83b66a9ca5ddd, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.493628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.527780 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.528093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.528383 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.532088 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1396/2807] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:03 2022] PUT /identity/v3/projects/ecd21a4bb6494ff5b219037d559e0805/users/7e49cd16847c4a46afd83b66a9ca5ddd/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:03.589243 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] MFA Rules not processed for user `7e49cd16847c4a46afd83b66a9ca5ddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:03.590394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:03.688799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1412/2808] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:03.693538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.801016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19baad61-810d-405f-878c-90d6b252ed86 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.806362 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.807048 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.807324 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.807599 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.808282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.810811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.851177 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1397/2809] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:47:03 2022] POST /identity/v3/projects => generated 385 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:03.855231 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:03.878260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253f9296-c3b0-4793-aecf-45bf747bb51d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:03.955830 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:03.957276 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:03.957965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:03.957965 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:03.959952 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:03.961402 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:03.990569 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:04.002158 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1413/2810] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:47:03 2022] POST /identity/v3/users => generated 299 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:47:04.006251 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:04.021735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1cc5ad-01d0-41e4-94f2-e5d68e1104a9 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:04.026975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:04.027823 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.028233 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.028598 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.029347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:04.031095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:04.055653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1398/2811] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:04 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:04.056658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:04.081942 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52a94be-0db9-4718-8c84-9c9f27c42960 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:04.104043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:04.104645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.104928 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.105202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] PATH_INFO: `/v3/projects/81c0857d8f674aa5917ec14dfcf60ebb/users/186e9274a7f54f13964eb03274a71fbd/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.126843 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81c0857d8f674aa5917ec14dfcf60ebb, user_id=186e9274a7f54f13964eb03274a71fbd, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:04.127517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:04.148466 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1414/2812] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:04 2022] PUT /identity/v3/projects/81c0857d8f674aa5917ec14dfcf60ebb/users/186e9274a7f54f13964eb03274a71fbd/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:04.152146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:04.166135 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb22833-961f-4e82-8c40-42f76762d958 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:04.230492 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:04.274461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab226b1-2c27-44f8-aaef-9981486acbbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:04.275145 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab226b1-2c27-44f8-aaef-9981486acbbd admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.275403 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab226b1-2c27-44f8-aaef-9981486acbbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.275747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab226b1-2c27-44f8-aaef-9981486acbbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.276319 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab226b1-2c27-44f8-aaef-9981486acbbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:04.277926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab226b1-2c27-44f8-aaef-9981486acbbd admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:04.293042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1399/2813] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:04 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:04.295260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:04.307834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc77d70-9de1-4b30-b23f-768f41e57d11 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:04.315657 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:04.317040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.317550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.318033 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] PATH_INFO: `/v3/projects/81c0857d8f674aa5917ec14dfcf60ebb/users/186e9274a7f54f13964eb03274a71fbd/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.320568 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=81c0857d8f674aa5917ec14dfcf60ebb, user_id=186e9274a7f54f13964eb03274a71fbd, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:04.323443 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:04.345152 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1415/2814] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:04 2022] PUT /identity/v3/projects/81c0857d8f674aa5917ec14dfcf60ebb/users/186e9274a7f54f13964eb03274a71fbd/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:04.350529 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.350843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.351147 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.409986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] MFA Rules not processed for user `186e9274a7f54f13964eb03274a71fbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:04.411332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:04.580354 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1400/2815] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:04 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:04.593058 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:04.596185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:04.718814 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d2afaa-985d-4634-9721-3f2723bb7d1d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:04.722979 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ad9395-9775-4ae4-9724-b6d530fb82fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:04.723481 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ad9395-9775-4ae4-9724-b6d530fb82fd service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.723863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ad9395-9775-4ae4-9724-b6d530fb82fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.724244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ad9395-9775-4ae4-9724-b6d530fb82fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.737233 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ad9395-9775-4ae4-9724-b6d530fb82fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:04.739197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ad9395-9775-4ae4-9724-b6d530fb82fd service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:04.777400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1416/2816] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:04.798173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:04.879140 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf0fd5b-3bfe-4bf6-a9b1-d2d24d08ee08 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:04.896022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:04.896022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:04.896022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:04.896022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] PATH_INFO: `/v3/users/7e49cd16847c4a46afd83b66a9ca5ddd` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:04.896917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e49cd16847c4a46afd83b66a9ca5ddd)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:04.900126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:04.979563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e49cd16847c4a46afd83b66a9ca5ddd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:04.991696 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e49cd16847c4a46afd83b66a9ca5ddd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.006714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:05.011410 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e49cd16847c4a46afd83b66a9ca5ddd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.012542 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e49cd16847c4a46afd83b66a9ca5ddd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.023869 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e49cd16847c4a46afd83b66a9ca5ddd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.032388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1401/2817] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:04 2022] DELETE /identity/v3/users/7e49cd16847c4a46afd83b66a9ca5ddd => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:05.038219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:05.038219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:05.038763 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:05.064275 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:05.066364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:05.160217 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1417/2818] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:05.180139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:05.181909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:05.273478 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17581f85-889e-4210-aa06-feb9cbe86c06 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:05.277579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:05.278193 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:05.278474 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:05.278749 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:05.290310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:05.291980 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:05.315699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1402/2819] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:05.658131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:05.659755 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:05.679853 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6808b18f-5b6a-4af9-b364-aa03df91e7a1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:05.689639 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:05.690570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:05.690959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:05.691394 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] PATH_INFO: `/v3/projects/ecd21a4bb6494ff5b219037d559e0805` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:05.693642 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ecd21a4bb6494ff5b219037d559e0805)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:05.701131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:05.778550 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ecd21a4bb6494ff5b219037d559e0805', 'request_id': 'req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.788868 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ecd21a4bb6494ff5b219037d559e0805', 'request_id': 'req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.792315 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:05.801040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ecd21a4bb6494ff5b219037d559e0805', 'request_id': 'req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.815834 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] The token cache is being invalidate because project ecd21a4bb6494ff5b219037d559e0805 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:05.816272 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.817343 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.823218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:05.823803 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1418/2820] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:05 2022] DELETE /identity/v3/projects/ecd21a4bb6494ff5b219037d559e0805 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:05.833533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd80110-c489-4103-a992-2dfb6c0668dc service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:05.905293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:05.906235 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:05.907498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:05.907498 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] PATH_INFO: `/v3/users/186e9274a7f54f13964eb03274a71fbd` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:05.909028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=186e9274a7f54f13964eb03274a71fbd)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:05.911120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:05.970506 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '186e9274a7f54f13964eb03274a71fbd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:05.984907 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '186e9274a7f54f13964eb03274a71fbd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.000008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '186e9274a7f54f13964eb03274a71fbd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.004895 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '186e9274a7f54f13964eb03274a71fbd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.026651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-f3c431e5-bf3c-4954-ad9d-7a036a5ed2ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '186e9274a7f54f13964eb03274a71fbd'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.038953 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1403/2821] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:05 2022] DELETE /identity/v3/users/186e9274a7f54f13964eb03274a71fbd => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:06.399363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:06.400850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:06.416708 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97094f9-b4c9-4810-8ed3-dae2e35a0ba1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:06.510879 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:06.511730 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:06.512147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:06.512520 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] PATH_INFO: `/v3/projects/81c0857d8f674aa5917ec14dfcf60ebb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:06.513847 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81c0857d8f674aa5917ec14dfcf60ebb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:06.516590 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:06.582621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81c0857d8f674aa5917ec14dfcf60ebb', 'request_id': 'req-5047fd6d-aec3-4177-9d33-62f34d885ec1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.595488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81c0857d8f674aa5917ec14dfcf60ebb', 'request_id': 'req-5047fd6d-aec3-4177-9d33-62f34d885ec1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.615781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81c0857d8f674aa5917ec14dfcf60ebb', 'request_id': 'req-5047fd6d-aec3-4177-9d33-62f34d885ec1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.632813 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] The token cache is being invalidate because project 81c0857d8f674aa5917ec14dfcf60ebb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:06.633183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.634032 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5047fd6d-aec3-4177-9d33-62f34d885ec1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:06.638910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1419/2822] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:06 2022] DELETE /identity/v3/projects/81c0857d8f674aa5917ec14dfcf60ebb => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:06.640623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:06.641553 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:06.641829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:06.667286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:06.684778 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:06.688418 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:06.769664 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1404/2823] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:06.777666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:06.777949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:06.778232 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:06.802740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:06.804031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:06.819218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:06.826756 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1420/2824] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:06.840176 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02e76ab-c431-4372-80f6-d08586cc61dc None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:06.849223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:06.850708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:06.851829 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:06.852199 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:06.852932 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:47:06.853304 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:06.855051 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:06.873008 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1405/2825] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:47:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 09 04:47:06.877884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:06.887826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2cd3c6-dd94-4294-a73e-43d37503b3fa None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:06.893037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:06.893744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:06.894091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:06.894440 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:06.895189 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:06.897408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:06.903573 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:06.905629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:06.931000 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1421/2826] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:47:06 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:06.933666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.016973 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=8,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:07.037338 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4235c-d92a-46ac-a20c-eee481ac4547 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.043099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.043864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.044246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.044620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.045364 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.048713 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.061421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6d53e4-30fc-42e5-9f2e-6587f5f39107 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.080948 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1422/2827] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:47:06 2022] POST /identity/v3/users => generated 290 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:07.084856 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.098711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a63d147-457a-46e0-a15e-7f230012bb53 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.104597 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.105238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.105516 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.105797 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.106490 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.108269 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.135622 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1423/2828] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:07 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:07.136855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.147310 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.147749 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00463319-7d52-4001-83dc-0fce805e33d6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.148059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.148502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.148915 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.150849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:07.153653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.154257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.154543 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.154829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.155910 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.158026 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.165025 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.166682 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3779558-78fc-4c72-a2ee-22174a4a0909 service placement] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.171594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1424/2829] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:07 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:07.176255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.179901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1406/2830] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:07.192053 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1313d9-b901-415e-9022-2ead209fbab1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.198869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.199595 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.199885 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.200165 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] PATH_INFO: `/v3/projects/a669cff8861444e1a8474cf3d2ad2bc7/users/48998c3c3ff94d878c47b6d2f4f187a0/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.220815 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a669cff8861444e1a8474cf3d2ad2bc7, user_id=48998c3c3ff94d878c47b6d2f4f187a0, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.223721 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.249464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1425/2831] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:07 2022] PUT /identity/v3/projects/a669cff8861444e1a8474cf3d2ad2bc7/users/48998c3c3ff94d878c47b6d2f4f187a0/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:47:07.262249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.262540 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.262833 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.277658 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:07.298590 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] MFA Rules not processed for user `48998c3c3ff94d878c47b6d2f4f187a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:07.299786 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:07.408595 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1407/2832] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:07 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:07.412023 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.493071 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1167dd74-48ff-4ab9-99e8-966fc9e654a5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.500162 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.500940 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.501325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.501714 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.502470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.504647 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.546632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1426/2833] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:47:07 2022] POST /identity/v3/projects => generated 381 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:07.547345 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.560904 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d756996e-f85a-4a91-b2b8-3342b5e28d81 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.648217 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.649133 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.649524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.649799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.652311 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.652458 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.680313 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1408/2834] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:47:07 2022] POST /identity/v3/users => generated 297 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:07.684222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.695837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fa81d1-818a-4008-87c4-cad2938d7a4b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.707570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.707570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.707570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.707570 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.708043 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.710324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.724318 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1427/2835] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:07 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:47:07.728314 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.739653 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1044f-e4e8-436b-a5e7-5000f41f4e9f admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.744858 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.745820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.746294 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.746865 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] PATH_INFO: `/v3/projects/a485cb1c16484a5e853e4ba01e6075ec/users/510c0db3100340729888f20e37b4ebff/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.763486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a485cb1c16484a5e853e4ba01e6075ec, user_id=510c0db3100340729888f20e37b4ebff, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.766765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.792269 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1409/2836] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:07 2022] PUT /identity/v3/projects/a485cb1c16484a5e853e4ba01e6075ec/users/510c0db3100340729888f20e37b4ebff/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:07.796148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.808503 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1318fa6e-f14e-46a9-8340-62e9614cb37b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.883632 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e7206-78f7-433f-aa9b-7d957daf7c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.884266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e7206-78f7-433f-aa9b-7d957daf7c30 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.884545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e7206-78f7-433f-aa9b-7d957daf7c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.884819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e7206-78f7-433f-aa9b-7d957daf7c30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.885484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e7206-78f7-433f-aa9b-7d957daf7c30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.887159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e7206-78f7-433f-aa9b-7d957daf7c30 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.899534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1428/2837] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:07 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:47:07.904039 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:07.915397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6024b352-7501-4c4a-9f35-fb811238d973 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:07.920933 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:07.921780 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.922098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.922375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] PATH_INFO: `/v3/projects/a485cb1c16484a5e853e4ba01e6075ec/users/510c0db3100340729888f20e37b4ebff/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.924637 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a485cb1c16484a5e853e4ba01e6075ec, user_id=510c0db3100340729888f20e37b4ebff, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:07.927419 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:07.940482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1410/2838] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:07 2022] PUT /identity/v3/projects/a485cb1c16484a5e853e4ba01e6075ec/users/510c0db3100340729888f20e37b4ebff/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:07.953915 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:07.954197 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:07.954461 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:07.966069 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:07.994967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] MFA Rules not processed for user `510c0db3100340729888f20e37b4ebff`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:07.994967 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:08.088702 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1429/2839] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:07 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:08.100398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:08.102144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:08.168098 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:08.182528 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f60af-ef33-4341-806a-dec8aee855d0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:08.186507 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:08.187082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:08.187411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:08.188283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:08.204134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:08.205613 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:08.216326 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1411/2840] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:08 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:08.301468 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:08.314139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd96ce7-2056-4b89-9cfd-e08c2e4a5b08 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:08.324993 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f7c3b-00c6-4290-9ee1-e8eac47364bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:08.325987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f7c3b-00c6-4290-9ee1-e8eac47364bd service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:08.325987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f7c3b-00c6-4290-9ee1-e8eac47364bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:08.326329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f7c3b-00c6-4290-9ee1-e8eac47364bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:08.391238 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1f7c3b-00c6-4290-9ee1-e8eac47364bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:08.392582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1f7c3b-00c6-4290-9ee1-e8eac47364bd service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:08.401899 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1430/2841] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:08 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:08.717851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:08.721064 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa7df4-e499-462f-95c7-5020e6236d2e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:08.721556 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa7df4-e499-462f-95c7-5020e6236d2e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:08.721666 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa7df4-e499-462f-95c7-5020e6236d2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:08.742846 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e6fa7df4-e499-462f-95c7-5020e6236d2e None None] MFA Rules not processed for user `c163b2e8db43490196e93b5fe4d45378`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:08.744120 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e6fa7df4-e499-462f-95c7-5020e6236d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:08.786099 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b7cdd6-de5e-4575-be72-02d80a35b084 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:08.790634 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:08.791287 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:08.791570 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:08.791840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] PATH_INFO: `/v3/users/48998c3c3ff94d878c47b6d2f4f187a0` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:08.792960 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=48998c3c3ff94d878c47b6d2f4f187a0)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:08.794866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:08.801949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1431/2842] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:47:08.847029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48998c3c3ff94d878c47b6d2f4f187a0'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:08.855694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48998c3c3ff94d878c47b6d2f4f187a0'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:08.870486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48998c3c3ff94d878c47b6d2f4f187a0'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:08.871234 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48998c3c3ff94d878c47b6d2f4f187a0'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:08.879402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48998c3c3ff94d878c47b6d2f4f187a0'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:08.885332 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1412/2843] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:08 2022] DELETE /identity/v3/users/48998c3c3ff94d878c47b6d2f4f187a0 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:08.890142 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:08.890467 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:08.891621 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:08.911648 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:08.912886 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:08.940980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:08.979130 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1432/2844] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:47:08.989317 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:08.990746 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:09.049734 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348b35a9-f705-4bc6-8e3c-0b37f8d25fb2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:09.053427 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:09.054036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:09.054312 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:09.054582 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:09.065158 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:09.066844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:09.076486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1413/2845] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:09.387291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:09.387291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:09.403449 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548c016-16d6-4206-a076-b6322494e6e8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:09.417488 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:09.418658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:09.419070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:09.419438 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] PATH_INFO: `/v3/projects/a669cff8861444e1a8474cf3d2ad2bc7` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:09.420676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a669cff8861444e1a8474cf3d2ad2bc7)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:09.423727 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:09.475535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a669cff8861444e1a8474cf3d2ad2bc7', 'request_id': 'req-117cb097-8bf4-47bb-8899-0c870070d371'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.484237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a669cff8861444e1a8474cf3d2ad2bc7', 'request_id': 'req-117cb097-8bf4-47bb-8899-0c870070d371'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.491230 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a669cff8861444e1a8474cf3d2ad2bc7', 'request_id': 'req-117cb097-8bf4-47bb-8899-0c870070d371'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.504852 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] The token cache is being invalidate because project a669cff8861444e1a8474cf3d2ad2bc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:09.505250 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.506393 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.512090 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1433/2846] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:09 2022] DELETE /identity/v3/projects/a669cff8861444e1a8474cf3d2ad2bc7 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:09.515936 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:09.525866 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0da84b-1c82-4692-8bff-031056838c43 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:09.535892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:09.590115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:09.590758 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:09.591070 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:09.591480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] PATH_INFO: `/v3/users/510c0db3100340729888f20e37b4ebff` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:09.592480 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] RBAC: Authorizing `identity:delete_user(user_id=510c0db3100340729888f20e37b4ebff)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:09.595050 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:09.646849 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '510c0db3100340729888f20e37b4ebff'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.654409 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '510c0db3100340729888f20e37b4ebff'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.668424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '510c0db3100340729888f20e37b4ebff'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.669279 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '510c0db3100340729888f20e37b4ebff'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.677347 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-16aa321e-fc79-4233-a988-5e5247c8fffe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '510c0db3100340729888f20e37b4ebff'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:09.686981 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1414/2847] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:09 2022] DELETE /identity/v3/users/510c0db3100340729888f20e37b4ebff => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:47:09.968606 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:09.971667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:09.992984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117cb097-8bf4-47bb-8899-0c870070d371 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.054844 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.055513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.055796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.056070 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] PATH_INFO: `/v3/projects/a485cb1c16484a5e853e4ba01e6075ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.057202 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a485cb1c16484a5e853e4ba01e6075ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.059135 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.109574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a485cb1c16484a5e853e4ba01e6075ec', 'request_id': 'req-cef7e3a6-1d7c-42de-afbc-55e4921f3979'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:10.117054 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a485cb1c16484a5e853e4ba01e6075ec', 'request_id': 'req-cef7e3a6-1d7c-42de-afbc-55e4921f3979'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:10.118296 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:10.123453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a485cb1c16484a5e853e4ba01e6075ec', 'request_id': 'req-cef7e3a6-1d7c-42de-afbc-55e4921f3979'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:10.137696 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] The token cache is being invalidate because project a485cb1c16484a5e853e4ba01e6075ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:10.138145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:10.138829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-cef7e3a6-1d7c-42de-afbc-55e4921f3979 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:10.140412 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1434/2848] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:09 2022] DELETE /identity/v3/projects/a485cb1c16484a5e853e4ba01e6075ec => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:10.145975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.146471 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.146718 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.166069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:10.167239 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:10.227186 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1415/2849] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:10.236098 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.236381 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.236658 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.257235 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:10.258435 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:10.267148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1435/2850] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:10.271512 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.281441 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3cbdda-d13d-4e09-98dc-032dbdbfd8d1 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.286138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.286739 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.287078 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.287350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] PATH_INFO: `/v3/domains` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.288029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:47:10.288361 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.289965 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.302297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1416/2851] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:47:10.307577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.318887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fb4c9c-a960-4480-9c77-7d398b8c7d5e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.323644 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.324240 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.324521 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.324798 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.325432 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.327372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.357836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1436/2852] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:47:10 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:10.362059 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.371938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2415be3-7330-4fd7-b578-57a7bedfaceb admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.389909 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:10.431096 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.431635 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.431917 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.432192 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.432861 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.434788 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.460204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1417/2853] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:47:10 2022] POST /identity/v3/users => generated 287 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:10.464237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.475015 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523da20c-e5f0-40b6-ac1c-e357d46b5760 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.479711 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.480300 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.480577 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.480848 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.481482 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.483099 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.493017 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1437/2854] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:10.497026 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.509156 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a7e6f-949b-495f-aa3f-2405d89b9a6d admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.516448 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb2770-ac0e-4e5a-847e-f789ee06e95d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.517244 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb2770-ac0e-4e5a-847e-f789ee06e95d admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.517707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb2770-ac0e-4e5a-847e-f789ee06e95d admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.517987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb2770-ac0e-4e5a-847e-f789ee06e95d admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.518633 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fb2770-ac0e-4e5a-847e-f789ee06e95d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.520282 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fb2770-ac0e-4e5a-847e-f789ee06e95d admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.531971 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1418/2855] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:10.536524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.546946 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705cd298-6823-4765-ba71-4dae6033fb37 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.551682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.552334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.552613 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.552884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] PATH_INFO: `/v3/projects/aa45ab7a2d004b98b0120df34602c162/users/721326416c15443391d60582f2f41926/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.585895 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa45ab7a2d004b98b0120df34602c162, user_id=721326416c15443391d60582f2f41926, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.589291 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.635779 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1438/2856] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:10 2022] PUT /identity/v3/projects/aa45ab7a2d004b98b0120df34602c162/users/721326416c15443391d60582f2f41926/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:10.702602 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2decb8be-4d4f-478d-b49d-51efdf92dcbe None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.702892 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2decb8be-4d4f-478d-b49d-51efdf92dcbe None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.703733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2decb8be-4d4f-478d-b49d-51efdf92dcbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.733434 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-2decb8be-4d4f-478d-b49d-51efdf92dcbe None None] MFA Rules not processed for user `721326416c15443391d60582f2f41926`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:10.734728 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2decb8be-4d4f-478d-b49d-51efdf92dcbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:10.804031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1419/2857] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:10.815491 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:10.817511 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:10.886801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57245bb-8bfc-44f6-b483-055c2231e3e0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:10.890661 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:10.891827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.892108 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.892378 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.901286 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:10.902744 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:10.914114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1439/2858] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:10.932062 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c36b33-f13b-42b9-b78a-393b31e86147 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:10.932559 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c36b33-f13b-42b9-b78a-393b31e86147 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:10.932841 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c36b33-f13b-42b9-b78a-393b31e86147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:10.965979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-25c36b33-f13b-42b9-b78a-393b31e86147 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:10.967255 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-25c36b33-f13b-42b9-b78a-393b31e86147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:11.036033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1420/2859] 10.222.0.20 () {58 vars in 1009 bytes} [Fri Sep 9 04:47:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:11.050369 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:11.052299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:11.135667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887e899-a5ca-4284-86ed-346058baf80e service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:11.142469 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:11.143122 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:11.143398 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:11.143672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:11.153334 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:11.154957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:11.164496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1440/2860] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:47:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 09 04:47:13.931152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3395e76-b5d7-4e90-8102-0c61d0f9c28d None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:13.931784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3395e76-b5d7-4e90-8102-0c61d0f9c28d None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:13.931784 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3395e76-b5d7-4e90-8102-0c61d0f9c28d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:13.959974 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-a3395e76-b5d7-4e90-8102-0c61d0f9c28d None None] MFA Rules not processed for user `9ece0d7befa845c0ab2a33bfb10b7850`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:13.961274 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-a3395e76-b5d7-4e90-8102-0c61d0f9c28d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:13.982914 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:14.034510 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1421/2861] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:14.213185 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:14.228073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:14.339479 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd354f1-00d0-4af4-8b08-64202684a2be service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:14.346971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:14.346971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:14.346971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:14.346971 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] PATH_INFO: `/v3/users/721326416c15443391d60582f2f41926` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:14.348538 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] RBAC: Authorizing `identity:delete_user(user_id=721326416c15443391d60582f2f41926)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:14.350257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:14.440368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '721326416c15443391d60582f2f41926'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:14.454966 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '721326416c15443391d60582f2f41926'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:14.476329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '721326416c15443391d60582f2f41926'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:14.477750 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '721326416c15443391d60582f2f41926'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:14.494134 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '721326416c15443391d60582f2f41926'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:14.510014 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1441/2862] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:47:14 2022] DELETE /identity/v3/users/721326416c15443391d60582f2f41926 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:14.512060 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:14.512450 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:14.512820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:14.533708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:14.535138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:14.611134 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1422/2863] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:14.623485 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:14.626667 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:14.776781 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22922c5-daf2-4181-bfc4-60efcf19e093 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:14.781056 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28db7c-512e-4d8c-ad35-98b4e6129297 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:14.781676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28db7c-512e-4d8c-ad35-98b4e6129297 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:14.781962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28db7c-512e-4d8c-ad35-98b4e6129297 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:14.782246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28db7c-512e-4d8c-ad35-98b4e6129297 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:14.803636 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28db7c-512e-4d8c-ad35-98b4e6129297 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:14.805105 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28db7c-512e-4d8c-ad35-98b4e6129297 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:14.843963 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1442/2864] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:47:15.230123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:15.232652 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:15.278043 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4959569f-42be-4c96-a607-a40ceb2af350 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:15.286286 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:15.287706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:15.288159 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:15.288808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] PATH_INFO: `/v3/projects/aa45ab7a2d004b98b0120df34602c162` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:15.290328 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa45ab7a2d004b98b0120df34602c162)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:15.296464 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:15.395056 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa45ab7a2d004b98b0120df34602c162', 'request_id': 'req-49b12ccc-0172-4667-bbad-ab5b09de2924'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:15.409797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa45ab7a2d004b98b0120df34602c162', 'request_id': 'req-49b12ccc-0172-4667-bbad-ab5b09de2924'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:15.419352 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa45ab7a2d004b98b0120df34602c162', 'request_id': 'req-49b12ccc-0172-4667-bbad-ab5b09de2924'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:15.465438 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] The token cache is being invalidate because project aa45ab7a2d004b98b0120df34602c162 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:47:15.468810 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:15.469814 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-49b12ccc-0172-4667-bbad-ab5b09de2924 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:47:15.472660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1423/2865] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:47:15 2022] DELETE /identity/v3/projects/aa45ab7a2d004b98b0120df34602c162 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:15.477608 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:15.478961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:15.478961 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:15.514829 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:15.517112 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:15.548690 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:15.695845 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1443/2866] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:47:15.704411 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:15.704979 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:15.705432 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:15.747423 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:15.749466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:15.765941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1424/2867] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:47:15.770311 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:15.781455 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1edaa6-35fe-4635-bef3-680a9efe1134 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:15.787103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:15.788067 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:15.788414 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:15.789205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:15.790073 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:47:15.790451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:15.792420 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:15.810285 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1444/2868] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:47:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:47:15.815236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:15.825137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ead26-e22a-4368-9d91-69ca3dfa460e None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:15.830733 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:15.831647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:15.831647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:15.831924 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:15.832644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:15.834906 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:15.865031 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1425/2869] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:47:15 2022] POST /identity/v3/projects => generated 381 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:15.870037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:15.881422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a00c8a-582f-43b1-84d7-bdd26d4f107c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:15.953594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:15.954205 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:15.954480 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:15.954753 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:15.955429 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:15.957329 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:15.986788 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1445/2870] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:47:15 2022] POST /identity/v3/users => generated 291 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:47:15.988072 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:15.998804 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881ff94b-ec1e-4b52-a38b-71d7572582ff admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:16.007986 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:16.012597 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:16.012941 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:16.013215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:16.013930 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:16.016280 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:16.027090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1426/2871] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:15 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:16.032100 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:16.042701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8d6457-dda6-4e4b-a5bd-c41af1d5c8e5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:16.047541 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09733fe-9262-4b59-9955-850f05b295e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:16.048255 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09733fe-9262-4b59-9955-850f05b295e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:16.048553 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09733fe-9262-4b59-9955-850f05b295e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:16.048951 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09733fe-9262-4b59-9955-850f05b295e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:16.049841 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09733fe-9262-4b59-9955-850f05b295e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:16.051836 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09733fe-9262-4b59-9955-850f05b295e4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:16.065431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1446/2872] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:47:16.068747 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:16.079604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012e7de1-b38f-4c66-bbb5-836819536156 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:16.084461 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:16.085129 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:16.085413 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:16.085688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] PATH_INFO: `/v3/projects/e40e149178c6430bac065ed46db7dbf8/users/2703b4f1781e4f009d51f0929d6c555b/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:16.098034 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:16.116797 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e40e149178c6430bac065ed46db7dbf8, user_id=2703b4f1781e4f009d51f0929d6c555b, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:16.122575 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:16.142846 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1427/2873] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:47:16 2022] PUT /identity/v3/projects/e40e149178c6430bac065ed46db7dbf8/users/2703b4f1781e4f009d51f0929d6c555b/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:47:16.148147 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de1a7ec-1207-4498-8e12-7724f30c9c62 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:16.148458 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de1a7ec-1207-4498-8e12-7724f30c9c62 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:16.148759 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de1a7ec-1207-4498-8e12-7724f30c9c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:16.169194 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6de1a7ec-1207-4498-8e12-7724f30c9c62 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:16.171136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6de1a7ec-1207-4498-8e12-7724f30c9c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:16.252367 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1447/2874] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:16.263168 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:16.266170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:16.331735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfb8593-46db-40c7-afdf-e11d8818eccc admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:16.335834 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:16.336468 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:16.336765 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:16.337021 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:16.349420 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:16.350890 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:16.362594 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1428/2875] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:17.172990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880bb834-1e60-4eba-b7c3-d4cd19fe9a1e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:17.173809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880bb834-1e60-4eba-b7c3-d4cd19fe9a1e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:17.173809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880bb834-1e60-4eba-b7c3-d4cd19fe9a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:17.205826 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-880bb834-1e60-4eba-b7c3-d4cd19fe9a1e None None] MFA Rules not processed for user `2703b4f1781e4f009d51f0929d6c555b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:17.207499 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-880bb834-1e60-4eba-b7c3-d4cd19fe9a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:17.219257 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:17.309654 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1448/2876] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:17 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:17.322172 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:17.323735 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:17.393011 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c330576-1384-42db-87ab-3e50017dcb43 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:17.396910 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:17.397516 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:17.397819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:17.398123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:17.413262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:17.414617 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:17.429975 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1429/2877] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:17 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:18.896103 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:18.896662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:18.896662 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:18.916980 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] MFA Rules not processed for user `2703b4f1781e4f009d51f0929d6c555b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:47:18.928884 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:18.943630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1449/2878] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:47:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:47:27.425120 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=56,keystone:INSERT=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:28.911433 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=25 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:37.165627 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:37.175693 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d67f46-d800-4b13-b640-370af2b5d223 service cinder] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:37.179214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:37.179801 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:37.180077 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:37.180349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:37.182035 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:37.244864 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:37.246296 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5be8f9-6fed-46b0-8667-931199050554 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:37.255821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1430/2879] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:37 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:39.530722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:47:39.541764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb89dc7-3a98-4843-b080-7aa075cfd56c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:47:39.545757 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:47:39.546638 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:47:39.547132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:47:39.547410 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:47:39.549323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:47:39.619310 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:47:39.622025 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:47:39.635139 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1450/2880] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:47:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:47:47.251513 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=30 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:47:49.630689 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=30 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:48:25.229523 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:48:25.230036 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:48:25.230151 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:48:25.249638 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] MFA Rules not processed for user `2703b4f1781e4f009d51f0929d6c555b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:48:25.250820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:48:25.257608 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1431/2881] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:48:25 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:48:25.485175 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:48:25.493968 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742069d-c991-4a79-9102-83e880981783 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:48:25.497672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:48:25.498265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:48:25.498545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:48:25.498820 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:48:25.500604 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:48:25.510131 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:48:25.511582 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:48:25.519472 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1451/2882] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:48:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:48:35.244094 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:48:35.515870 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:48:57.604882 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:48:57.615236 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684cc5b7-2cfe-4935-9c7a-9f1cb6643658 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:48:57.618800 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:48:57.619428 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:48:57.619709 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:48:57.619984 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:48:57.621707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:48:57.703223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:48:57.704654 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:48:57.725533 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1432/2883] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:48:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:49:07.722993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=43 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:22.685557 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:22.699191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2d78f1-5408-4c82-9a02-6b8235b49c9d service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:22.704020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:22.704722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:22.705008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:22.705283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] PATH_INFO: `/v3/users/2703b4f1781e4f009d51f0929d6c555b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:22.706422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2703b4f1781e4f009d51f0929d6c555b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:22.708409 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:22.754620 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2703b4f1781e4f009d51f0929d6c555b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:22.762031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2703b4f1781e4f009d51f0929d6c555b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:22.774063 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2703b4f1781e4f009d51f0929d6c555b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:22.774710 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2703b4f1781e4f009d51f0929d6c555b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:22.781795 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-216f9c1a-eff7-4654-829f-0c7e20b45024 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2703b4f1781e4f009d51f0929d6c555b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:22.788900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1452/2884] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:49:22 2022] DELETE /identity/v3/users/2703b4f1781e4f009d51f0929d6c555b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:22.963769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:22.977130 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c90dc4-1423-4ed5-bafa-769243f34206 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.036851 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.037481 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.037756 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.038052 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] PATH_INFO: `/v3/projects/e40e149178c6430bac065ed46db7dbf8` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.039211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e40e149178c6430bac065ed46db7dbf8)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.041102 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.090995 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e40e149178c6430bac065ed46db7dbf8', 'request_id': 'req-b757ec7d-b25e-405d-ba2c-1dacc9c62191'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:23.101212 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e40e149178c6430bac065ed46db7dbf8', 'request_id': 'req-b757ec7d-b25e-405d-ba2c-1dacc9c62191'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:23.108863 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e40e149178c6430bac065ed46db7dbf8', 'request_id': 'req-b757ec7d-b25e-405d-ba2c-1dacc9c62191'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:23.121396 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] The token cache is being invalidate because project e40e149178c6430bac065ed46db7dbf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:49:23.121690 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:23.122397 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-b757ec7d-b25e-405d-ba2c-1dacc9c62191 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:23.127215 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1433/2885] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:49:22 2022] DELETE /identity/v3/projects/e40e149178c6430bac065ed46db7dbf8 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:23.129222 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.129671 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.129955 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.154591 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:23.155822 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:23.225193 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1453/2886] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:49:23.233509 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.234003 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.234398 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.258211 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:23.260146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:23.270496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1434/2887] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:49:23.274653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.285922 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bde0bb-8bcc-4188-8b8e-f30022372a60 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.291091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.291704 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.291986 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.292339 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.293031 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 09 04:49:23.293340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.295451 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.322540 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1454/2888] 10.222.0.20 () {58 vars in 1153 bytes} [Fri Sep 9 04:49:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 09 04:49:23.327200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.344843 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c57907-706e-4e2a-ad8b-16bb5931c405 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.351309 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.351931 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.352266 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.352545 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.353203 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.355160 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.439388 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1435/2889] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:49:23 2022] POST /identity/v3/projects => generated 373 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:49:23.443855 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.455700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4a87d-37d5-4b38-ab34-de3499530288 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.559024 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.561020 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.561534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.562047 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.562917 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.566371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.577742 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:23.603645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1455/2890] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:49:23 2022] POST /identity/v3/users => generated 287 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:49:23.608247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.633146 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5120c4b-6236-422c-855e-5eb9d96406d4 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.642170 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.643022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.644014 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.644708 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.645080 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.646783 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.662179 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1436/2891] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:49:23 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:49:23.663453 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.675574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2715d813-f6e6-46ea-a379-e9a3f189f4ec admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.685950 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf405ab-0d70-4d3b-818b-abe5ab294a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.686580 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf405ab-0d70-4d3b-818b-abe5ab294a40 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.688947 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf405ab-0d70-4d3b-818b-abe5ab294a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.689184 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf405ab-0d70-4d3b-818b-abe5ab294a40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.689863 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf405ab-0d70-4d3b-818b-abe5ab294a40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.695969 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf405ab-0d70-4d3b-818b-abe5ab294a40 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.718566 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1456/2892] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:49:23 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:49:23.720002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.732231 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5240fc-acfe-4844-b1f8-57371b90d165 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.738446 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.739673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.739673 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.740308 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] PATH_INFO: `/v3/projects/78feec3c37b84399ba54e70b4e5a1e08/users/deab6d2c9e1f429ca593f92669deab9c/roles/807749e5ca90423999feeb8a7727599c` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.758604 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78feec3c37b84399ba54e70b4e5a1e08, user_id=deab6d2c9e1f429ca593f92669deab9c, role_id=807749e5ca90423999feeb8a7727599c)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:23.761408 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:23.780439 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1437/2893] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:49:23 2022] PUT /identity/v3/projects/78feec3c37b84399ba54e70b4e5a1e08/users/deab6d2c9e1f429ca593f92669deab9c/roles/807749e5ca90423999feeb8a7727599c => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:23.785288 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c3f75e-bcab-4c77-9be5-51ada6c03250 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.785575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c3f75e-bcab-4c77-9be5-51ada6c03250 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.785849 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c3f75e-bcab-4c77-9be5-51ada6c03250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:23.807764 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a3c3f75e-bcab-4c77-9be5-51ada6c03250 None None] MFA Rules not processed for user `f2b9f22768104befbb8f56575b4c786b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:23.809148 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a3c3f75e-bcab-4c77-9be5-51ada6c03250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:23.883920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1457/2894] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:49:23.915113 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:23.917191 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:23.941351 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:23.989825 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a0140a-2d0f-4b4c-b384-6cfe0e3579f0 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:23.996440 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:23.997114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:23.997393 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:23.997844 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:24.009101 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:24.010556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:24.022016 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1438/2895] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:49:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:49:28.712563 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:28.713464 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:28.713960 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:28.747957 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] MFA Rules not processed for user `deab6d2c9e1f429ca593f92669deab9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:28.749283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:28.836993 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:28.841949 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1458/2896] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:49:28.850057 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468b958-2397-4f60-bb8f-9b1e53b3dd5e service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:28.855557 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:28.856412 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:28.856697 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:28.856973 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:28.857615 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:28.859782 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:28.886353 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:28.897795 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1439/2897] 10.222.0.20 () {60 vars in 1153 bytes} [Fri Sep 9 04:49:28 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 09 04:49:28.899306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232b99ff-d896-492b-aa8f-fdf51bbf744c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:28.901935 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:28.981732 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:28.982787 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:28.983328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:28.983809 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:28.986316 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:28.988353 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6432e8-c93b-4660-8d12-294ed4282cb2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:28.993938 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:28.994644 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:28.995033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:28.995375 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] PATH_INFO: `/v3/users` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:28.996254 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:28.998502 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:29.014867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:29.038262 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1440/2898] 10.222.0.20 () {60 vars in 1144 bytes} [Fri Sep 9 04:49:28 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 09 04:49:29.042335 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:29.056138 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28804a4d-e6eb-4cf4-8e6d-4f4daf1116fc admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:29.062847 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:29.063663 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:29.064020 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:29.064363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:29.065152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:29.067032 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:29.083349 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1441/2899] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:49:29 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 09 04:49:29.084787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:29.088880 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:29.090513 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:29.095714 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b0e82d-3430-42a4-a3c3-ef67a60284c2 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:29.103660 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:29.104632 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:29.105095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:29.105579 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] PATH_INFO: `/v3/projects/4a643d4d0b2f41f09c3d90c27fb14b41/users/02eceb9d07ad41fdab759d537511c4f6/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:29.106802 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1459/2900] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:49:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:49:29.123256 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a643d4d0b2f41f09c3d90c27fb14b41, user_id=02eceb9d07ad41fdab759d537511c4f6, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:29.126216 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:29.143547 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1442/2901] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:49:29 2022] PUT /identity/v3/projects/4a643d4d0b2f41f09c3d90c27fb14b41/users/02eceb9d07ad41fdab759d537511c4f6/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:29.147533 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:29.162653 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096ddad-b28c-452b-b938-f98f1faf4b79 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:29.250283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d875ae5-960d-4fa0-a1f2-a4a7857b12b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:29.251046 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d875ae5-960d-4fa0-a1f2-a4a7857b12b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:29.251434 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d875ae5-960d-4fa0-a1f2-a4a7857b12b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:29.251850 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d875ae5-960d-4fa0-a1f2-a4a7857b12b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:29.252656 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d875ae5-960d-4fa0-a1f2-a4a7857b12b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:29.254400 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d875ae5-960d-4fa0-a1f2-a4a7857b12b6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:29.272314 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1460/2902] 10.222.0.20 () {58 vars in 1122 bytes} [Fri Sep 9 04:49:29 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 09 04:49:29.273366 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:29.284996 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f10cf16-ad33-48ce-b4df-75cb12c3dbfa admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:29.289926 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:29.290581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] REQUEST_METHOD: `PUT` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:29.290856 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:29.291196 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] PATH_INFO: `/v3/projects/4a643d4d0b2f41f09c3d90c27fb14b41/users/02eceb9d07ad41fdab759d537511c4f6/roles/a5062e61497c4d40b35fe67595be3e2e` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:29.293214 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a643d4d0b2f41f09c3d90c27fb14b41, user_id=02eceb9d07ad41fdab759d537511c4f6, role_id=a5062e61497c4d40b35fe67595be3e2e)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:29.295918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:29.306730 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1443/2903] 10.222.0.20 () {60 vars in 1483 bytes} [Fri Sep 9 04:49:29 2022] PUT /identity/v3/projects/4a643d4d0b2f41f09c3d90c27fb14b41/users/02eceb9d07ad41fdab759d537511c4f6/roles/a5062e61497c4d40b35fe67595be3e2e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:31.577943 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0ed01-5536-4811-a4ae-24e92f5692e4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:31.579052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0ed01-5536-4811-a4ae-24e92f5692e4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:31.579340 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0ed01-5536-4811-a4ae-24e92f5692e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:31.637191 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bde0ed01-5536-4811-a4ae-24e92f5692e4 None None] MFA Rules not processed for user `719e5e2221e04ce0aa7f303b2ea84b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:31.638408 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bde0ed01-5536-4811-a4ae-24e92f5692e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:31.717805 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1461/2904] 10.222.0.20 () {58 vars in 967 bytes} [Fri Sep 9 04:49:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:49:31.785249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:31.787071 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:31.855798 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:31.927107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac90063a-080c-488f-917b-f3e4a3cb7373 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:31.935487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a835e652fcc43ab923c9c6ab434d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:31.936271 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:31.936655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:31.937023 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:31.949967 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:31.952190 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:31.982744 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1444/2905] 10.222.0.20 () {60 vars in 1384 bytes} [Fri Sep 9 04:49:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 09 04:49:34.831040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff81e1e-3773-4892-811d-e3fa89c39a7b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:34.831909 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff81e1e-3773-4892-811d-e3fa89c39a7b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:34.832172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff81e1e-3773-4892-811d-e3fa89c39a7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:34.871173 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dff81e1e-3773-4892-811d-e3fa89c39a7b None None] MFA Rules not processed for user `02eceb9d07ad41fdab759d537511c4f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:34.872681 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dff81e1e-3773-4892-811d-e3fa89c39a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:34.889460 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:34.976471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1462/2906] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:34 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:49:34.991019 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:34.993028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:35.100090 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adc8586-7e3a-458b-b4fe-1c380aa7a574 service swift] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:35.106629 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:35.107556 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:35.108009 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:35.108281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:35.191611 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:35.194404 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:35.224382 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1445/2907] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:49:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 09 04:49:36.898999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:36.899495 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:36.899583 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:36.944237 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] MFA Rules not processed for user `deab6d2c9e1f429ca593f92669deab9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:36.945978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:36.966748 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1463/2908] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:49:40.920661 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:40.922748 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:40.986901 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=100 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:41.005006 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0061e8ba-1e9b-410f-a5d9-05b37fc9da4b service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:41.010118 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:41.010768 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:41.011566 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:41.011710 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] PATH_INFO: `/v3/users/c163b2e8db43490196e93b5fe4d45378` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:41.013015 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c163b2e8db43490196e93b5fe4d45378)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:41.015486 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:41.069729 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c163b2e8db43490196e93b5fe4d45378'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.084082 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c163b2e8db43490196e93b5fe4d45378'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.106769 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c163b2e8db43490196e93b5fe4d45378'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.107552 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c163b2e8db43490196e93b5fe4d45378'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.117110 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c163b2e8db43490196e93b5fe4d45378'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.124223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1446/2909] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:49:40 2022] DELETE /identity/v3/users/c163b2e8db43490196e93b5fe4d45378 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:41.313864 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:41.325457 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd917b0-3b15-40a0-aae6-80d4a9a7cd0e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:41.394008 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:41.394542 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:41.394827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:41.395127 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] PATH_INFO: `/v3/projects/9931b1bca2464db7bcb90b91567cab28` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:41.396242 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9931b1bca2464db7bcb90b91567cab28)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:41.398166 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:41.447900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9931b1bca2464db7bcb90b91567cab28', 'request_id': 'req-5816fa6e-89b6-446a-81de-205c95de88c9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.455893 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9931b1bca2464db7bcb90b91567cab28', 'request_id': 'req-5816fa6e-89b6-446a-81de-205c95de88c9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.461790 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9931b1bca2464db7bcb90b91567cab28', 'request_id': 'req-5816fa6e-89b6-446a-81de-205c95de88c9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.472801 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] The token cache is being invalidate because project 9931b1bca2464db7bcb90b91567cab28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:49:41.473093 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.473736 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.475278 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1464/2910] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:49:41 2022] DELETE /identity/v3/projects/9931b1bca2464db7bcb90b91567cab28 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:41.479293 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:41.480699 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:41.494194 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca3665-4cf8-4484-8d47-57696eb06d3b admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:41.551524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:41.552069 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:41.552343 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:41.552616 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] PATH_INFO: `/v3/users/dc1ff8a54ae34493bc88cb3ca209f2fa` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:41.553731 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc1ff8a54ae34493bc88cb3ca209f2fa)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:41.555707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:41.603121 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc1ff8a54ae34493bc88cb3ca209f2fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.610300 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc1ff8a54ae34493bc88cb3ca209f2fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.620538 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:41.623247 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc1ff8a54ae34493bc88cb3ca209f2fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.624250 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc1ff8a54ae34493bc88cb3ca209f2fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.633248 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-65032d75-8db0-407b-aeb3-dd645992f696 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc1ff8a54ae34493bc88cb3ca209f2fa'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.639033 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1447/2911] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:49:41 2022] DELETE /identity/v3/users/dc1ff8a54ae34493bc88cb3ca209f2fa => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:41.816701 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:41.818522 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:41.834246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5816fa6e-89b6-446a-81de-205c95de88c9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:41.864551 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:41.899545 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:41.900179 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:41.900465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:41.900740 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] PATH_INFO: `/v3/projects/045033350adb4269aac1665b7ff23237` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:41.901872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=045033350adb4269aac1665b7ff23237)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:41.903837 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:41.957145 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '045033350adb4269aac1665b7ff23237', 'request_id': 'req-e890dd1b-2139-4060-a594-31c2ac9484c4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.964422 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '045033350adb4269aac1665b7ff23237', 'request_id': 'req-e890dd1b-2139-4060-a594-31c2ac9484c4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.970283 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '045033350adb4269aac1665b7ff23237', 'request_id': 'req-e890dd1b-2139-4060-a594-31c2ac9484c4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.981206 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] The token cache is being invalidate because project 045033350adb4269aac1665b7ff23237 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:49:41.981496 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.982124 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:49:41.983664 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1465/2912] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:49:41 2022] DELETE /identity/v3/projects/045033350adb4269aac1665b7ff23237 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:49:51.635581 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:DELETE=1,keystone:SELECT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:51.978260 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=39,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:49:52.054114 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:52.054402 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:52.054680 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:52.076922 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] MFA Rules not processed for user `9ece0d7befa845c0ab2a33bfb10b7850`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:49:52.078148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:52.135395 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1448/2913] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:49:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 09 04:49:52.989177 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:52.991136 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:53.063215 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e890dd1b-2139-4060-a594-31c2ac9484c4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:53.069060 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:53.069682 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:53.069959 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:53.070229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:53.072036 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:53.084594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:53.086186 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:53.097091 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1466/2914] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:49:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:49:55.048049 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:49:55.050703 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:55.129421 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca07bdca-e6f1-4d85-8d9d-3370259404f9 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:49:55.134276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:49:55.134887 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:49:55.135200 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:49:55.135472 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:49:55.137227 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:49:55.148115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:49:55.149563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:49:55.171157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1449/2915] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:49:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:50:03.092328 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=32 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:50:05.155103 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=51 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:50:07.617800 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:50:07.619371 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:07.694524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79bf100-c0f1-4bb4-8279-9196f11fb121 service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:50:07.701239 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:50:07.705872 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:50:07.706164 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:50:07.706475 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:50:07.708359 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:07.721038 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:50:07.723204 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:50:07.734319 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1467/2916] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:50:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:50:11.782819 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:50:11.803028 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58780a94-d00e-4165-88b6-cb0d24053750 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:50:11.812281 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:50:11.813007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:50:11.813173 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:50:11.813524 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:50:11.815905 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:11.828888 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:50:11.830496 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:50:11.842131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1450/2917] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:50:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:50:12.595683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:50:12.597323 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:12.614002 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a55e156-8823-4c1a-b36e-3eaa2d8a5107 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:50:12.618631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:50:12.619279 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:50:12.619562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:50:12.619827 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:50:12.621579 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:12.698862 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:50:12.700325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:50:12.710984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1468/2918] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:50:12 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:50:14.510785 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:50:14.512350 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:14.526155 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7febdb5-a96f-4abf-b5cd-934e2511138d service nova] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:50:14.532424 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dd190c922964d51af263442650b427d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:50:14.533029 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:50:14.533307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:50:14.534068 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:50:14.535850 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:50:14.549002 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:50:14.551609 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706bc782-a7d0-4f21-80cb-3529a6ed2dac service nova] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:50:14.563116 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1451/2919] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:50:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:50:22.706891 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=64 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:50:24.558257 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=12 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:51:54.953631 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:51:54.955308 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:51:55.033252 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d677427-2b02-4002-9c36-ba8b9504e8c6 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:51:55.036819 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a8add98306764181b5ea1e8a1190bbde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:51:55.037418 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:51:55.037700 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:51:55.037976 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:51:55.040219 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:51:55.050792 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:51:55.052246 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:51:55.061752 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1469/2920] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:51:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:51:58.675466 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] REQUEST_METHOD: `POST` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:51:58.676000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:51:58.676000 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:51:58.695482 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.auth.core [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] MFA Rules not processed for user `9ece0d7befa845c0ab2a33bfb10b7850`. Rule list: `[]` (Enabled: `True`). {{(pid=81451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 09 04:51:58.696688 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:51:58.703519 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1452/2921] 10.222.0.20 () {58 vars in 958 bytes} [Fri Sep 9 04:51:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 09 04:51:59.025778 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:51:59.035132 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e59f6-4ff4-4631-8466-eb5eef2653de service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:51:59.038695 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:51:59.039325 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:51:59.039611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:51:59.039887 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:51:59.041574 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:51:59.051052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:51:59.052465 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:51:59.060945 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1470/2922] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:51:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:52:08.689707 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=3 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:52:09.057456 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=41 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:52:11.881243 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:52:11.883889 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:52:11.899272 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646b5e1b-c9cb-48b5-80a6-ef478112e675 None None] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:52:11.904218 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:52:11.904821 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:52:11.905123 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:52:11.905363 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:52:11.907107 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:52:11.919706 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:52:11.921623 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:52:11.933701 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1453/2923] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:52:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:52:21.927306 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:52:25.327022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:52:25.336518 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3088c-b79f-4a35-ac55-ce48f00d9180 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:52:25.340363 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:52:25.340992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:52:25.341350 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:52:25.341693 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:52:25.343517 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:52:25.354022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:52:25.355575 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:52:25.364042 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1471/2924] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:52:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:52:26.474860 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:52:26.484763 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c948bd63-5eaa-4aca-8e89-26b90f07d2b0 service placement] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:52:26.488364 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:52:26.488962 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:52:26.489242 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:52:26.489515 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:52:26.491666 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:52:26.505628 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:52:26.507131 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:52:26.519152 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1454/2925] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:52:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:52:35.360266 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:52:36.513563 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=6 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:53:58.415183 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:53:58.424074 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f3128-e28b-4302-a2f7-4dedde1619cf service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:53:58.427990 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:53:58.428594 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:53:58.428878 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:53:58.429159 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:53:58.431722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:53:58.507720 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:53:58.509195 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:53:58.519484 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1472/2926] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:53:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:54:00.599793 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:00.613260 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a89e0b-df76-4836-a89a-15e9e4242af8 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:00.618526 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4c501a6bee5b4f3a8973f727f9ef10cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:00.619435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:00.619928 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:00.620429 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:00.623126 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:00.641862 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:00.643704 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:00.660358 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1455/2927] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:54:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:54:01.971978 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:01.987585 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91b07c6-2231-423c-9370-3225fccba03a service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:01.996772 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:01.997342 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:01.997611 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:01.997984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:02.000180 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:02.020450 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:02.023626 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:02.035115 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1473/2928] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:54:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:54:05.791491 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:05.794657 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:05.811387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a89541-5adf-4336-b1fc-aa3135b85dd9 service neutron] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:05.817204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:05.817918 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:05.818276 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:05.818651 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] PATH_INFO: `/v3/users/9ece0d7befa845c0ab2a33bfb10b7850` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:05.820154 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ece0d7befa845c0ab2a33bfb10b7850)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:05.822223 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:05.883307 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ece0d7befa845c0ab2a33bfb10b7850'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:05.891139 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ece0d7befa845c0ab2a33bfb10b7850'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:05.904787 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ece0d7befa845c0ab2a33bfb10b7850'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:05.905561 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ece0d7befa845c0ab2a33bfb10b7850'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:05.913115 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ece0d7befa845c0ab2a33bfb10b7850'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:05.920022 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1456/2929] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:54:05 2022] DELETE /identity/v3/users/9ece0d7befa845c0ab2a33bfb10b7850 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:06.155560 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:06.167900 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e4d109-bfce-41d9-85a1-59f9df27f139 service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:06.244372 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:06.245172 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:06.245368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:06.245709 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] PATH_INFO: `/v3/projects/ea4c285aae544f47a91407ff79414128` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:06.246869 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea4c285aae544f47a91407ff79414128)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:06.249040 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:06.306676 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea4c285aae544f47a91407ff79414128', 'request_id': 'req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.319670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea4c285aae544f47a91407ff79414128', 'request_id': 'req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.327037 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea4c285aae544f47a91407ff79414128', 'request_id': 'req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.340867 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] The token cache is being invalidate because project ea4c285aae544f47a91407ff79414128 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:54:06.341331 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.342181 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.343962 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1474/2930] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:54:06 2022] DELETE /identity/v3/projects/ea4c285aae544f47a91407ff79414128 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:06.348144 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:06.349750 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:06.363518 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e823ac46-c4e3-468c-8e67-5c302276a790 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:06.426237 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:06.427204 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:06.427655 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:06.428095 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] PATH_INFO: `/v3/users/611d6e38f9d9486f8d95d2ca2d692819` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:06.429297 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=611d6e38f9d9486f8d95d2ca2d692819)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:06.431323 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:06.483684 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '611d6e38f9d9486f8d95d2ca2d692819'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.492157 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '611d6e38f9d9486f8d95d2ca2d692819'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.497961 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:54:06.506820 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '611d6e38f9d9486f8d95d2ca2d692819'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.507630 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '611d6e38f9d9486f8d95d2ca2d692819'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.515585 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '611d6e38f9d9486f8d95d2ca2d692819'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.521592 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1457/2931] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:54:06 2022] DELETE /identity/v3/users/611d6e38f9d9486f8d95d2ca2d692819 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:06.729999 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:06.731799 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:06.748021 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cb1f2c-dd0e-4944-9e12-6bbb26d71652 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:06.772505 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:54:06.832510 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:06.833010 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:06.833289 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:06.833562 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] PATH_INFO: `/v3/projects/c2ef1a079e644830876e5d6423987db9` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:06.834907 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2ef1a079e644830876e5d6423987db9)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:06.837333 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:06.901417 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2ef1a079e644830876e5d6423987db9', 'request_id': 'req-5642af6d-a406-4540-a721-2826523ff4f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.910859 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2ef1a079e644830876e5d6423987db9', 'request_id': 'req-5642af6d-a406-4540-a721-2826523ff4f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.918722 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2ef1a079e644830876e5d6423987db9', 'request_id': 'req-5642af6d-a406-4540-a721-2826523ff4f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.931052 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] The token cache is being invalidate because project c2ef1a079e644830876e5d6423987db9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:54:06.931471 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.932229 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:06.943187 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1475/2932] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:54:06 2022] DELETE /identity/v3/projects/c2ef1a079e644830876e5d6423987db9 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:07.128387 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:07.130201 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:07.236618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ad2c1c-f339-4207-bd92-be2deeddc5a6 admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:07.305618 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:07.306246 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] REQUEST_METHOD: `GET` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:07.306527 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:07.306799 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:07.308624 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:07.321477 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:07.323148 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:07.333999 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1458/2933] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:54:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:54:10.311373 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:10.312920 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:10.378743 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5642af6d-a406-4540-a721-2826523ff4f4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:10.383368 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:10.383987 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:10.384264 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:10.384534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] PATH_INFO: `/v3/users/deab6d2c9e1f429ca593f92669deab9c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:10.385672 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] RBAC: Authorizing `identity:delete_user(user_id=deab6d2c9e1f429ca593f92669deab9c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:10.387630 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:10.433706 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:54:10.436857 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'deab6d2c9e1f429ca593f92669deab9c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.445330 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'deab6d2c9e1f429ca593f92669deab9c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.457534 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'deab6d2c9e1f429ca593f92669deab9c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.458201 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'deab6d2c9e1f429ca593f92669deab9c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.465683 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'deab6d2c9e1f429ca593f92669deab9c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.471535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1476/2934] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:54:10 2022] DELETE /identity/v3/users/deab6d2c9e1f429ca593f92669deab9c => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 09 04:54:10.642137 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:10.653751 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4e69f0-865c-46c3-ab15-877b0988e759 service placement] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:10.711717 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:10.712342 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:10.712692 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:10.712966 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] PATH_INFO: `/v3/projects/78feec3c37b84399ba54e70b4e5a1e08` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:10.714081 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] RBAC: Authorizing `identity:delete_project(project_id=78feec3c37b84399ba54e70b4e5a1e08)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:10.716042 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:10.727089 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:54:10.762224 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78feec3c37b84399ba54e70b4e5a1e08', 'request_id': 'req-2bba6dd9-839b-469a-9457-5a17e8a8a3da'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.769500 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78feec3c37b84399ba54e70b4e5a1e08', 'request_id': 'req-2bba6dd9-839b-469a-9457-5a17e8a8a3da'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.774840 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78feec3c37b84399ba54e70b4e5a1e08', 'request_id': 'req-2bba6dd9-839b-469a-9457-5a17e8a8a3da'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.785694 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] The token cache is being invalidate because project 78feec3c37b84399ba54e70b4e5a1e08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:54:10.785987 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.786647 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.788125 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1459/2935] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:54:10 2022] DELETE /identity/v3/projects/78feec3c37b84399ba54e70b4e5a1e08 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:10.792619 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:10.794022 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:10.807584 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191735f7-a4e7-486f-96a8-45fc204b4090 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:10.869992 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:10.870625 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:10.870903 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:10.871224 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] PATH_INFO: `/v3/users/02eceb9d07ad41fdab759d537511c4f6` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:10.873431 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02eceb9d07ad41fdab759d537511c4f6)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:10.876324 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:10.925270 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02eceb9d07ad41fdab759d537511c4f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.935306 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02eceb9d07ad41fdab759d537511c4f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.947470 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02eceb9d07ad41fdab759d537511c4f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.948249 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02eceb9d07ad41fdab759d537511c4f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.955645 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02eceb9d07ad41fdab759d537511c4f6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:10.961641 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1477/2936] 10.222.0.20 () {58 vars in 1224 bytes} [Fri Sep 9 04:54:10 2022] DELETE /identity/v3/users/02eceb9d07ad41fdab759d537511c4f6 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:11.131238 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Authenticating user token {{(pid=81451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:54:11.132808 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.fernet_utils [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:54:11.147505 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bba6dd9-839b-469a-9457-5a17e8a8a3da admin admin] Validating token access rules against request {{(pid=81451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:54:11.208249 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2b9f22768104befbb8f56575b4c786b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a501f44f388545198539783ea38ef108', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:54:11.208879 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:54:11.209162 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:54:11.209435 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] PATH_INFO: `/v3/projects/4a643d4d0b2f41f09c3d90c27fb14b41` {{(pid=81451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:54:11.210551 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a643d4d0b2f41f09c3d90c27fb14b41)` {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:54:11.213007 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] RBAC: Authorization granted {{(pid=81451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:54:11.261891 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a643d4d0b2f41f09c3d90c27fb14b41', 'request_id': 'req-e8f68334-bda2-43ca-abe0-3a3795e58ea7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:11.270487 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a643d4d0b2f41f09c3d90c27fb14b41', 'request_id': 'req-e8f68334-bda2-43ca-abe0-3a3795e58ea7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:11.277549 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a643d4d0b2f41f09c3d90c27fb14b41', 'request_id': 'req-e8f68334-bda2-43ca-abe0-3a3795e58ea7'} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:11.288752 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] The token cache is being invalidate because project 4a643d4d0b2f41f09c3d90c27fb14b41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 09 04:54:11.289045 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:11.290283 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG keystone.notifications [None req-e8f68334-bda2-43ca-abe0-3a3795e58ea7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 09 04:54:11.291811 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: [pid: 81451|app: 0|req: 1460/2937] 10.222.0.20 () {58 vars in 1233 bytes} [Fri Sep 9 04:54:11 2022] DELETE /identity/v3/projects/4a643d4d0b2f41f09c3d90c27fb14b41 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 09 04:54:20.958114 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=14,keystone:SELECT=66,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:54:21.285489 dv-cinder-856002-3-iscsi devstack@keystone.service[81451]: DEBUG dbcounter [-] [81451] Writing DB stats keystone:SELECT=70,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 09 04:57:15.846670 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 09 04:57:15.848160 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:57:15.984524 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6025d0c-d4b3-4a35-b6a5-3a338f290951 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 09 04:57:16.042360 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef39450c46b43a5bed04645689610f6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6ac32ce05067416fbc8af4a0e7da4cfe', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 09 04:57:16.042984 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 09 04:57:16.043265 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 09 04:57:16.043535 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 09 04:57:16.045241 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 09 04:57:16.056244 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 09 04:57:16.057796 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf799e3-0d52-4da0-ac1c-8d60df0d1a09 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 09 04:57:16.068078 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1478/2938] 10.222.0.20 () {60 vars in 1365 bytes} [Fri Sep 9 04:57:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 09 04:57:26.063299 dv-cinder-856002-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}